Re: [gentoo-dev] Re: [RFC] Adding potentially questionable license AcePerl-Indemnity

[Richard Yao]

On 4/23/20 3:31 AM, Ulrich Mueller wrote:
>> On Thu, 23 Apr 2020, Kent Fredric wrote:
> 
>> I've just discovered dev-perl/Ace has some fun questionable licensing
>> which includes a lovely indemnity clause, which had previously gone
>> unnoticed, and it stipulates additional requests for research
>> publications, which is not something mentioned in any license currently
>> in tree other than Tinker
> 
>> Following is the entire body of the license I plan to put in
>> licenses/AcePerl-Indemnity ( name chosen to specifically alert people
>> tempted to accept this license that Indemnification is an important
>> part they should actually read )
> 
>> Current advice also says that due to the terms of this license, we have
>> to RESTRICT="mirror" this as well, unless the Trustees want to sign off
>> on potentially indemnifying CSHL
> 
>> Also following up with CPAN because as its *currently* mirrored on
>> CPAN, and has been mirrored there for at *least* 12 years, its
>> potentially in a legal situation as well.
> 
>> ( But that's the fault of the uploader if true, because you can't
>> upload anything to CPAN without mirroring being something you didn't
>> expect )
> 
>> Once this license is added, the plan is to rework Ace-*.ebuild to be under
> 
>> LICENSE="|| ( Artistic GPL-1+ ) AcePerl-Indemnity" 
> 
>> Upstream: https://metacpan.org/source/LDS/AcePerl-1.92/DISCLAIMER.txt
> 
> The important words are:
> "This library is free software; you can redistribute it and/or modify it
> under the same terms as Perl itself."
> which makes it simply LICENSE="|| ( Artistic GPL-1+ )", and of course,
> no mirror restriction is needed.
> 
> The rest is simply an additional warranty disclaimer. I wouldn't even
> see it as part of the license, because it is about usage of the
> software, not about its distribution.

The language then goes on to add additional terms, so it isn't only
under the terms of the perl license. There are two things that worry me
here.

The first is the indemnification clause. Indemnification in the US is
like an insurance policy. If it were to go to court over something
covered, those who agreed to provide indemnification must pay the legal
expenses of those were taken to court. People have lost in courts for
things as small as commas. I am not a lawyer, but I think this needs
additional attention.

The second is the attribution clause. While it seems silly, this has two
possible interpretations:

1. A disclaimer.
2. A solid requirement for people of a certain field of endeavor imposed
by the license and enforced by either legal or extralegal means.

In the case of the former, it is fine, but in the case of the latter,
this is enough to render the license non-free on two grounds:

1. Clause 5 of the OSD that prohibits discrimination against persons or
groups:

https://opensource.org/osd-annotated

2. The dissident test that we borrow from Debian regarding "excess"
distribution:

https://wiki.gentoo.org/wiki/License_groups#When_is_a_license_a_free_software_license.3F

If this were worded as a reminder, this would not be a problem, but it
seems like it could be considered a non-optional request by virtue of
copyright being restrictive unless explicit permission is granted. If
the language had been along the lines of a recommendation or a reminder
of the requirement that credit be provided as required by the academic
community for academic integrity, then it would have been fine. For
example, something like this would have worked:

> It is recommended that if publications result from research using this
SOFTWARE, CSHL be acknowledged and/or credit be given to CSHL
scientists, as is scientifically appropriate.

However, this is what was written:

> If publications result from research using this SOFTWARE, we ask that
CSHL be acknowledged and/or credit be given to CSHL scientists, as
scientifically appropriate.

Requests aren't always optional, so this might be construable as the
license imposing a requirement. It is a minor point compared to the
indemnity clause, but I think it merits additional scrutiny.

>
> As always: IANAL, TINLA.
> 
> Ulrich
> 




signature.asc
Description: OpenPGP digital signature

Re: ext4 readdir performance - was Re: [gentoo-dev] New distfile mirror layout

[Richard Yao]

> On Oct 23, 2019, at 7:48 PM, Richard Yao  wrote:
> 
> On 10/22/19 2:51 AM, Jaco Kroon wrote:
>> Hi All,
>> 
>> 
>>> On 2019/10/21 18:42, Richard Yao wrote:
>>> 
>>> If we consider the access frequency, it might actually not be that
>>> bad. Consider a simple example with 500 files and two directory
>>> buckets. If we have 250 in each, then the size of the directory is
>>> always 250. However, if 50 files are accessed 90% of the time, then
>>> putting 450 into one directory and that 50 into another directory, we
>>> end up with the performance of the O(n) directory lookup being
>>> consistent with there being only 90 files in each directory.
>>> 
>>> I am not sure if we should be discarding all other considerations to
>>> make changes to benefit O(n) directory lookup filesystems, but if we
>>> are, then the hashing approach is not necessarily the best one. It is
>>> only the best when all files are accessed with equal frequency, which
>>> would be an incorrect assumption. A more human friendly approach might
>>> still be better. I doubt that we have the data to determine that though.
>>> 
>>> Also, another idea is to use a cheap hash function (e.g. fletcher) and
>>> just have the mirrors do the hashing behind the scenes. Then we would
>>> have the best of both worlds.
>> 
>> 
>> Experience:
>> 
>> ext4 sucks at targeting name lookups without dir_index feature (O(n)
>> lookups - scans all entries in the folder).  With dir_index readdir
>> performance is crap.  Pick your poison I guess.  Most of our larger
>> filesystems (2TB+, but especially the 80TB+ ones) we've reverted to
>> disabling dir_index as the benefit is outweighed by the crappy readdir()
>> and glob() performance.
> My read of the ext4 disk layout documentation is that the read operation
> should work mostly the same way, except with a penalty from reading
> larger directories caused by the addition of the tree's metadata and
> from having more partially filled blocks:
> 
> https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Directory_Entries
> 
> The code itself is the same traversal code:
> 
> https://github.com/torvalds/linux/blob/v5.3/fs/ext4/dir.c#L106
> 
> However, a couple of things stand out to me here at a glance:
> 
> 1. `cond_resched()` adds scheduler delay for no apparent reason.
> `cond_resched()` is meant to be used in places where we could block
> excessively on non-PREEMPT kernels, but this doesn't strike me as one of
> those places. The fact that we block on disk on uncached reads naturally
> serves the same purpose, so an explicit rescheduling point here is
> redundant. PREEMPT kernels should perform better in readdir() on ext4 by
> virtue of making `cond_resched()` a no-op.
I just realized that the way that I worded this could be confusing, so please 
allow me to clarify what I meant. cond_resched() is meant for when a kernel 
thread will tie up a CPU for a long period of time. Blocking on disk will cause 
the CPU to be released to another thread. When we do not block on disk, this 
operation is quick. There is no good reason for putting cond_resched() here as 
far as I can tell.
> 2. read-ahead is implemented in a way that appears to be over-reading
> the directory whenever the needed information is not cached. This is
> technically read-ahead, although it is not a great way of doing it. A
> much better way to do this would be to pipeline `readdir()` by
> initiating asynchronous read operations in anticipation of future reads.
> 
> Both of thse should affect both variants of ext4's directories, but the
> penalties I mentioned earlier mean that the dir_index variant would be
> affected more.
> 
> If you have a way to benchmark things, a simple idea to evaluate would
> be deleting the `cond_resched()` line. If we had data showing an
> improvement, I would be happy to send a small one-line patch deleting
> the line to Ted to get the change into mainline.
The more I think about this, the more absurd having cond_resched() here seems 
to me. I think I will sit on it for a few days. If it still seems absurd to me 
after sitting on it, I’ll send Ted a patch to delete that line with the remark 
that the use of cond_resched() is redundant with blocking on disk.
>> There doesn't seem to be a real specific tip-over point, and it seems to
>> depend a lot on RAM availability and harddrive speed (obviously).  So if
>> dentries gets cached, disk speeds becomes less of an issue.  However, on
>> large folders (where I typically use 10k as a value for large based on
>> "gut feeling" and "unquantifiable experience" and &qu

ext4 readdir performance - was Re: [gentoo-dev] New distfile mirror layout

[Richard Yao]

On 10/22/19 2:51 AM, Jaco Kroon wrote:
> Hi All,
> 
> 
> On 2019/10/21 18:42, Richard Yao wrote:
>>
>> If we consider the access frequency, it might actually not be that
>> bad. Consider a simple example with 500 files and two directory
>> buckets. If we have 250 in each, then the size of the directory is
>> always 250. However, if 50 files are accessed 90% of the time, then
>> putting 450 into one directory and that 50 into another directory, we
>> end up with the performance of the O(n) directory lookup being
>> consistent with there being only 90 files in each directory.
>>
>> I am not sure if we should be discarding all other considerations to
>> make changes to benefit O(n) directory lookup filesystems, but if we
>> are, then the hashing approach is not necessarily the best one. It is
>> only the best when all files are accessed with equal frequency, which
>> would be an incorrect assumption. A more human friendly approach might
>> still be better. I doubt that we have the data to determine that though.
>>
>> Also, another idea is to use a cheap hash function (e.g. fletcher) and
>> just have the mirrors do the hashing behind the scenes. Then we would
>> have the best of both worlds.
> 
> 
> Experience:
> 
> ext4 sucks at targeting name lookups without dir_index feature (O(n)
> lookups - scans all entries in the folder).  With dir_index readdir
> performance is crap.  Pick your poison I guess.  Most of our larger
> filesystems (2TB+, but especially the 80TB+ ones) we've reverted to
> disabling dir_index as the benefit is outweighed by the crappy readdir()
> and glob() performance.
My read of the ext4 disk layout documentation is that the read operation
should work mostly the same way, except with a penalty from reading
larger directories caused by the addition of the tree's metadata and
from having more partially filled blocks:

https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Directory_Entries

The code itself is the same traversal code:

https://github.com/torvalds/linux/blob/v5.3/fs/ext4/dir.c#L106

However, a couple of things stand out to me here at a glance:

1. `cond_resched()` adds scheduler delay for no apparent reason.
`cond_resched()` is meant to be used in places where we could block
excessively on non-PREEMPT kernels, but this doesn't strike me as one of
those places. The fact that we block on disk on uncached reads naturally
serves the same purpose, so an explicit rescheduling point here is
redundant. PREEMPT kernels should perform better in readdir() on ext4 by
virtue of making `cond_resched()` a no-op.

2. read-ahead is implemented in a way that appears to be over-reading
the directory whenever the needed information is not cached. This is
technically read-ahead, although it is not a great way of doing it. A
much better way to do this would be to pipeline `readdir()` by
initiating asynchronous read operations in anticipation of future reads.

Both of thse should affect both variants of ext4's directories, but the
penalties I mentioned earlier mean that the dir_index variant would be
affected more.

If you have a way to benchmark things, a simple idea to evaluate would
be deleting the `cond_resched()` line. If we had data showing an
improvement, I would be happy to send a small one-line patch deleting
the line to Ted to get the change into mainline.

> There doesn't seem to be a real specific tip-over point, and it seems to
> depend a lot on RAM availability and harddrive speed (obviously).  So if
> dentries gets cached, disk speeds becomes less of an issue.  However, on
> large folders (where I typically use 10k as a value for large based on
> "gut feeling" and "unquantifiable experience" and "nothing scientific at
> all") I find that even with lots of RAM two consecutive ls commands
> remains terribly slow. Switch off dir_index and that becomes an order of
> magnitude faster.
> 
> I don't have a great deal of experience with XFS, but on those systems
> where we do it's generally on a VM, and perceivably (again, not
> scientific) our experience has been that it feels slower.  Again, not
> scientific, just perception.
> 
> I'm in support for the change.  This will bucket to 256 folders and
> should have a reasonably even split between folders.  If required a
> second layer could be introduced by using the 3rd and 4th digits of the
> hash for a second layer.  Any hash should be fine, it really doesn't
> need to be cryptographically strong, it just needs to provide a good
> spread and be really fast.  Generally a hash table should have a prime
> number of buckets to assist with hash bias, but frankly, that's over
> complicating the situation here.
> 
> I also agree with others that it used to be easy to get distfiles as and

Re: [gentoo-dev] New distfile mirror layout

[Richard Yao]

> On Oct 20, 2019, at 2:51 AM, Michał Górny  wrote:
> 
> On Sat, 2019-10-19 at 19:24 -0400, Joshua Kinard wrote:
>>> On 10/18/2019 09:41, Michał Górny wrote:
>>> Hi, everybody.
>>> 
>>> It is my pleasure to announce that yesterday (EU) evening we've switched
>>> to a new distfile mirror layout.  Users will be switching to the new
>>> layout either as they upgrade Portage to 2.3.77 or -- if they upgraded
>>> already -- as their caches expire (24hrs).
>>> 
>>> The new layout is mostly a bow towards mirror admins, for some of whom
>>> having a 6+ files in a single directory have been a problem. 
>>> However, I suppose some of you also found e.g. the directory index
>>> hardly usable due to its size.
>>> 
>>> Throughout a transitional period (whose exact length hasn't been decided
>>> yet), both layouts will be available.  Afterwards, the old layout will
>>> be removed from mirrors.  This has a few implications:
>>> 
>>> 1. Users who don't upgrade their package managers in time will lose
>>> the ability of fetching from Gentoo mirrors.  This shouldn't be that
>>> much of a problem given that the core software needed to upgrade Portage
>>> should all have reliable upstream SRC_URIs.
>>> 
>>> 2. mirror://gentoo/file URIs will stop working.  While technically you
>>> could use mirror://gentoo/XX/file, I'd rather recommend finally
>>> discarding its usage and moving distfiles to devspace.
>>> 
>>> 3. Directly fetching files from distfiles.gentoo.org will become
>>> a little harder.  To fetch a distfile named 'foo-1.tar.gz', you'd have
>>> to use something like:
>>> 
>>> $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2
>>> 1b
>>> $ wget http://distfiles.gentoo.org/distfiles/1b/foo-1.tar.gz
>>> ...
>>> 
>>> 
>>> Alternatively, you can:
>>> 
>>> $ wget http://distfiles.gentoo.org/distfiles/INDEX
>>> 
>>> and grep for the right path there.  This INDEX is also a more
>>> lightweight alternative to HTML indexes generated by the servers.
>>> 
>>> 
>>> If you're interested in more background details and some plots, see [1].
>>> 
>>> [1] 
>>> https://dev.gentoo.org/~mgorny/articles/improving-distfile-mirror-structure.html
>>> 
>> 
>> So the answer I didn't really see directly stated here is, where do new
>> distfiles need to go //now//?  E.g., if on woodpecker, I currently cp a
>> distfile to /space/distfiles-local.  What is the new directory I need to
>> use?  And if mirror://gentoo/${FOO} is going away, for the new distfiles
>> target, what would be the applicable prefix to use?
>> 
>> Directly using devspace seems like a bad idea, IMHO.  Once long ago, we all
>> got chastised for doing exactly that.  Too much possibility of fragmentation
>> as devs retire or package maintainership changes hands.
> 
> Today you get chastised for using /space/distfiles-local and not
> following policy changes.  The devmanual states that it's deprecated
> since at least 2011, and talks of using d.g.o [1].
> 
>> I looked at the whitepaper'ish-like writeup, and I kinda don't like using a
>> hash-based naming scheme on the new distfiles layout.  I really kind prefer
>> breaking the directories up based on the first letter of the distfiles in
>> question, factoring case-sensitivity in (so you'd have 52 top-level
>> directories for A-Z and a-z, plus 10 more for 0-9).  Under each of those
>> directories, additional subdirectories for the next few letters (say,
>> letters 2-3).  Yes, this leads to some orphan cases where a distfile might
>> live on its own, but from a direct navigation standpoint, it's easy to find
>> for someone browsing the distfiles server and easy to predict where a
>> distfile is at.
>> 
>> No math, statistical analysis, or deep-rooted knowledge of filesystems
>> behind that paragraph.  Just a plain old unfiltered opinion.  Sometimes, I
>> need to go get a distfile off the Gentoo mirrors, and being able to quickly
>> find it in the mirror root is great.  Having to do hash calculations to work
>> out the file path will be *really* annoying.
> 
> Your solution still doesn't solve the problem of having 8k-24k files
> in a single directory, even if you use 7 letters of prefix.  So it just
> creates a lot of tiny directory noise for no practical gain.
> 
> [1] 
> https://devmanual.gentoo.org/general-concepts/mirrors/index.html#suitable-download-hosts

If we consider the access frequency, it might actually not be that bad. 
Consider a simple example with 500 files and two directory buckets. If we have 
250 in each, then the size of the directory is always 250. However, if 50 files 
are accessed 90% of the time, then putting 450 into one directory and that 50 
into another directory, we end up with the performance of the O(n) directory 
lookup being consistent with there being only 90 files in each directory.

I am not sure if we should be discarding all other considerations to make 
changes to benefit O(n) directory lookup filesystems, but if we are, then the 
hashing approach is not necessarily the best one. It is only the 

Re: [gentoo-dev] New distfile mirror layout

[Richard Yao]

> On Oct 19, 2019, at 4:03 PM, Michał Górny  wrote:
> 
> On Sat, 2019-10-19 at 15:26 -0400, Richard Yao wrote:
>>>> On Oct 18, 2019, at 9:10 PM, Richard Yao  wrote:
>>> 
>>> 
>>>>> On Oct 18, 2019, at 4:49 PM, Michał Górny  wrote:
>>>> On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote:
>>>>>>>>>> On Oct 18, 2019, at 9:42 AM, Michał Górny  wrote:
>>>>>>>>> Hi, everybody.
>>>>>>>>> It is my pleasure to announce that yesterday (EU) evening we've 
>>>>>>>>> switched
>>>>>>>>> to a new distfile mirror layout.  Users will be switching to the new
>>>>>>>>> layout either as they upgrade Portage to 2.3.77 or -- if they upgraded
>>>>>>>>> already -- as their caches expire (24hrs).
>>>>>>>>> The new layout is mostly a bow towards mirror admins, for some of whom
>>>>>>>>> having a 6+ files in a single directory have been a problem.
>>>>>>>>> However, I suppose some of you also found e.g. the directory index
>>>>>>>>> hardly usable due to its size.
>>>>> This sounds like a filesystem issue. Do we know which filesystems are 
>>>>> suffering?
>>>>> ZFS should be fine. I believe ext2/ext3 have problems with this many 
>>>>> files. ext4 is probably okay, but don’t quote me on that.
>>>> Ext2, VFAT and NTFS were mentioned on the bug [1], though I suppose this
>>>> may apply only to older ntfs versions.  NFS has been mentioned too.
>>> 
>>> ext2 and vfat are not surprises to me (outside of the idea that anyone 
>>> would use them for a mirror). NTFS and NFS are though.
>>>> However, just because modern filesystems can handle them efficiently, it
>>>> doesn't mean having directories that huge comes with zero cost.
>>> While I am okay with the change, what do you mean when you say that having 
>>> huge directories does not come with zero cost?
>>> 
>>> Filesystems with O(1) directory lookups like ZFS would probably be hurt by 
>>> this, but the impact should be negligible. Filesystems with O(log n) 
>>> directory lookups would see faster directory lookups.
>>> 
>>> Outside of directory lookups, this could speed up up searches and sort 
>>> operations when listing everything with just about any filesystem 
>>> benefiting from the improvement.
>>> 
>>> Listing directories on such filesystems should not benefit from this unless 
>>> you are using ls where the default behavior is to sort the directory 
>>> contents (which is where the improvement when sorting comes into play). The 
>>> need to sort the directory contents by default keeps ls from displaying 
>>> anything until it has scanned the entire directory. The asymptotic 
>>> complexity of a fast comparison based sort improves in this situation from 
>>> O(nlogn) to O(nlog(n/b)) provided that you sort each subdirectory 
>>> independently. A further speed up could be obtained by doing multithreading 
>>> to parallelize the sort operations.
>> I read your original email late at night and I misread the description of 
>> how this works.
>> 
>> At an initial glance, I thought we were doing a prefix approach (with the 
>> caveat that buckets are unbalanced). In reality, we are doing a 
>> cryptographic hash of the filenames.
>> 
>> That would keep all buckets balanced, which gives the best directory lookup 
>> times on O(log n) lookup filesystems, but I think there is something to be 
>> gained from using the less optimal approach of using filename prefixes:
>> 
>> * some regex searches on distfiles can be accelerated
>> * generating a sorted list of all distfiles becomes asymptotically faster
>> * it is easy for a user to find all versions of a given distfile
>> * no need to calculate a cryptographic hash
>> 
>> I realize that I am late to propose it, but could we consider a switch to 
>> this alternative arrangement?
> 
> No, we can't.  Please read either the original discussion on the bug, or
> the linked article.  It's explained in detail why this won't work.
Alright. I am convinced. Thanks.
> 
> -- 
> Best regards,
> Michał Górny
> 

Re: [gentoo-dev] New distfile mirror layout

[Richard Yao]

> On Oct 18, 2019, at 9:10 PM, Richard Yao  wrote:
> 
> 
>>> On Oct 18, 2019, at 4:49 PM, Michał Górny  wrote:
>> On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote:
>>>>>>>> On Oct 18, 2019, at 9:42 AM, Michał Górny  wrote:
>>>>>>> Hi, everybody.
>>>>>>> It is my pleasure to announce that yesterday (EU) evening we've switched
>>>>>>> to a new distfile mirror layout.  Users will be switching to the new
>>>>>>> layout either as they upgrade Portage to 2.3.77 or -- if they upgraded
>>>>>>> already -- as their caches expire (24hrs).
>>>>>>> The new layout is mostly a bow towards mirror admins, for some of whom
>>>>>>> having a 6+ files in a single directory have been a problem.
>>>>>>> However, I suppose some of you also found e.g. the directory index
>>>>>>> hardly usable due to its size.
>>> This sounds like a filesystem issue. Do we know which filesystems are 
>>> suffering?
>>> ZFS should be fine. I believe ext2/ext3 have problems with this many files. 
>>> ext4 is probably okay, but don’t quote me on that.
>> Ext2, VFAT and NTFS were mentioned on the bug [1], though I suppose this
>> may apply only to older ntfs versions.  NFS has been mentioned too.
> 
> ext2 and vfat are not surprises to me (outside of the idea that anyone would 
> use them for a mirror). NTFS and NFS are though.
>> However, just because modern filesystems can handle them efficiently, it
>> doesn't mean having directories that huge comes with zero cost.
> While I am okay with the change, what do you mean when you say that having 
> huge directories does not come with zero cost?
> 
> Filesystems with O(1) directory lookups like ZFS would probably be hurt by 
> this, but the impact should be negligible. Filesystems with O(log n) 
> directory lookups would see faster directory lookups.
> 
> Outside of directory lookups, this could speed up up searches and sort 
> operations when listing everything with just about any filesystem benefiting 
> from the improvement.
> 
> Listing directories on such filesystems should not benefit from this unless 
> you are using ls where the default behavior is to sort the directory contents 
> (which is where the improvement when sorting comes into play). The need to 
> sort the directory contents by default keeps ls from displaying anything 
> until it has scanned the entire directory. The asymptotic complexity of a 
> fast comparison based sort improves in this situation from O(nlogn) to 
> O(nlog(n/b)) provided that you sort each subdirectory independently. A 
> further speed up could be obtained by doing multithreading to parallelize the 
> sort operations.
I read your original email late at night and I misread the description of how 
this works.

At an initial glance, I thought we were doing a prefix approach (with the 
caveat that buckets are unbalanced). In reality, we are doing a cryptographic 
hash of the filenames.

That would keep all buckets balanced, which gives the best directory lookup 
times on O(log n) lookup filesystems, but I think there is something to be 
gained from using the less optimal approach of using filename prefixes:

* some regex searches on distfiles can be accelerated
* generating a sorted list of all distfiles becomes asymptotically faster
* it is easy for a user to find all versions of a given distfile
* no need to calculate a cryptographic hash

I realize that I am late to propose it, but could we consider a switch to this 
alternative arrangement?

The bulk of the performance gain should be realized with either approach.

> Since I know someone will call me out on that comment, I will explain. Each 
> bucket has roughly n/b items in it where n is the total number and b is the 
> number of buckets. Sorting one bucket is O(n/b * log(n/b)). Loop to sort each 
> of the b buckets. The buckets are pre-sorted by prefix, so the result is now 
> sorted. You therefore get O(nlog(n/b)) time complexity out of an O(nlogn) 
> comparison sort on this very special case where you call it multiple times on 
> data that has been persorted by prefix into buckets.
> 
> Is there any other benefit to this or did I get everything?
> 
> By the way, it is offtopic for the thread, but it occurs to me that a hybrid 
> of radix sort and A comparison based sort could give us a general sorting 
> algorithm that is asymptotically faster than O(nlogn).
>> [1] https://bugs.gentoo.org/534528
>> --
>> Best regards,
>> Michał Górny

Re: [gentoo-dev] New distfile mirror layout

[Richard Yao]

> On Oct 19, 2019, at 2:17 AM, Michał Górny  wrote:
> 
> On Fri, 2019-10-18 at 21:09 -0400, Richard Yao wrote:
>>>> On Oct 18, 2019, at 4:49 PM, Michał Górny  wrote:
>>> 
>>> On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote:
>>>>>>>> On Oct 18, 2019, at 9:42 AM, Michał Górny  wrote:
>>>>>>> Hi, everybody.
>>>>>>> It is my pleasure to announce that yesterday (EU) evening we've switched
>>>>>>> to a new distfile mirror layout.  Users will be switching to the new
>>>>>>> layout either as they upgrade Portage to 2.3.77 or -- if they upgraded
>>>>>>> already -- as their caches expire (24hrs).
>>>>>>> The new layout is mostly a bow towards mirror admins, for some of whom
>>>>>>> having a 6+ files in a single directory have been a problem.
>>>>>>> However, I suppose some of you also found e.g. the directory index
>>>>>>> hardly usable due to its size.
>>>> This sounds like a filesystem issue. Do we know which filesystems are 
>>>> suffering?
>>>> ZFS should be fine. I believe ext2/ext3 have problems with this many 
>>>> files. ext4 is probably okay, but don’t quote me on that.
>>> 
>>> Ext2, VFAT and NTFS were mentioned on the bug [1], though I suppose this
>>> may apply only to older ntfs versions.  NFS has been mentioned too.
>> 
>> ext2 and vfat are not surprises to me (outside of the idea that anyone would 
>> use them for a mirror). NTFS and NFS are though.
> 
> Are you surprised that people use NTFS on Windows?  Or that they use
> local mirrors over NFS?  The latter still needs to be addressed
> separatel, provided that they mount it on DISTDIR.
I am surprised that it was an issue on NTFS because it uses B-trees. As for 
NFS, I had expected that to be more dependent on the local filesystem than on 
NFS itself. If it has a slowdown when used on a filesystem that had fast 
directory operations, that might be a bug.
> 
>>> However, just because modern filesystems can handle them efficiently, it
>>> doesn't mean having directories that huge comes with zero cost.
>> While I am okay with the change, what do you mean when you say that having 
>> huge directories does not come with zero cost?
>> 
>> Filesystems with O(1) directory lookups like ZFS would probably be hurt by 
>> this
> 
> O(1) or O(n)?
ZFS uses extendible hashing for its directories, so the data structure used is 
amortized O(1). You might consider it O(log n) due to the indirect tree 
traversal needed to find the direct block containing the hash table entry. With 
caching of indirect blocks, it should be amortized O(1) to find the direct 
block in practice as far as read IOs are considered. In addition, the base of 
the logarithm is 128 or 1024 depending on the pool feature flags.
> 
>> , but the impact should be negligible. Filesystems with O(log n) directory 
>> lookups would see faster directory lookups.
>> 
>> Outside of directory lookups, this could speed up up searches and sort 
>> operations when listing everything with just about any filesystem benefiting 
>> from the improvement.
>> 
>> Listing directories on such filesystems should not benefit from this unless 
>> you are using ls where the default behavior is to sort the directory 
>> contents (which is where the improvement when sorting comes into play). The 
>> need to sort the directory contents by default keeps ls from displaying 
>> anything until it has scanned the entire directory. The asymptotic 
>> complexity of a fast comparison based sort improves in this situation from 
>> O(nlogn) to O(nlog(n/b)) provided that you sort each subdirectory 
>> independently. A further speed up could be obtained by doing multithreading 
>> to parallelize the sort operations.
>> 
>> Since I know someone will call me out on that comment, I will explain. Each 
>> bucket has roughly n/b items in it where n is the total number and b is the 
>> number of buckets. Sorting one bucket is O(n/b * log(n/b)). Loop to sort 
>> each of the b buckets. The buckets are pre-sorted by prefix, so the result 
>> is now sorted. You therefore get O(nlog(n/b)) time complexity out of an 
>> O(nlogn) comparison sort on this very special case where you call it 
>> multiple times on data that has been persorted by prefix into buckets.
>> 
>> Is there any other benefit to this or did I get everything?
> 
> Listings for individual directories won't cause major pain to browsers
> anymore.  Not that there's much reason to do them.
That m

Re: [gentoo-dev] New distfile mirror layout

[Richard Yao]

> On Oct 18, 2019, at 4:49 PM, Michał Górny  wrote:
> 
> On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote:
>>>>>> On Oct 18, 2019, at 9:42 AM, Michał Górny  wrote:
>>>>> Hi, everybody.
>>>>> It is my pleasure to announce that yesterday (EU) evening we've switched
>>>>> to a new distfile mirror layout.  Users will be switching to the new
>>>>> layout either as they upgrade Portage to 2.3.77 or -- if they upgraded
>>>>> already -- as their caches expire (24hrs).
>>>>> The new layout is mostly a bow towards mirror admins, for some of whom
>>>>> having a 6+ files in a single directory have been a problem.
>>>>> However, I suppose some of you also found e.g. the directory index
>>>>> hardly usable due to its size.
>> This sounds like a filesystem issue. Do we know which filesystems are 
>> suffering?
>> ZFS should be fine. I believe ext2/ext3 have problems with this many files. 
>> ext4 is probably okay, but don’t quote me on that.
> 
> Ext2, VFAT and NTFS were mentioned on the bug [1], though I suppose this
> may apply only to older ntfs versions.  NFS has been mentioned too.

ext2 and vfat are not surprises to me (outside of the idea that anyone would 
use them for a mirror). NTFS and NFS are though.
> 
> However, just because modern filesystems can handle them efficiently, it
> doesn't mean having directories that huge comes with zero cost.
While I am okay with the change, what do you mean when you say that having huge 
directories does not come with zero cost?

Filesystems with O(1) directory lookups like ZFS would probably be hurt by 
this, but the impact should be negligible. Filesystems with O(log n) directory 
lookups would see faster directory lookups.

Outside of directory lookups, this could speed up up searches and sort 
operations when listing everything with just about any filesystem benefiting 
from the improvement.

Listing directories on such filesystems should not benefit from this unless you 
are using ls where the default behavior is to sort the directory contents 
(which is where the improvement when sorting comes into play). The need to sort 
the directory contents by default keeps ls from displaying anything until it 
has scanned the entire directory. The asymptotic complexity of a fast 
comparison based sort improves in this situation from O(nlogn) to O(nlog(n/b)) 
provided that you sort each subdirectory independently. A further speed up 
could be obtained by doing multithreading to parallelize the sort operations.

Since I know someone will call me out on that comment, I will explain. Each 
bucket has roughly n/b items in it where n is the total number and b is the 
number of buckets. Sorting one bucket is O(n/b * log(n/b)). Loop to sort each 
of the b buckets. The buckets are pre-sorted by prefix, so the result is now 
sorted. You therefore get O(nlog(n/b)) time complexity out of an O(nlogn) 
comparison sort on this very special case where you call it multiple times on 
data that has been persorted by prefix into buckets.

Is there any other benefit to this or did I get everything?

By the way, it is offtopic for the thread, but it occurs to me that a hybrid of 
radix sort and A comparison based sort could give us a general sorting 
algorithm that is asymptotically faster than O(nlogn).
> 
> [1] https://bugs.gentoo.org/534528
> 
> -- 
> Best regards,
> Michał Górny

Re: [gentoo-dev] New distfile mirror layout

[Richard Yao]

> On Oct 18, 2019, at 9:42 AM, Michał Górny  wrote:
> 
> Hi, everybody.
> 
> It is my pleasure to announce that yesterday (EU) evening we've switched
> to a new distfile mirror layout.  Users will be switching to the new
> layout either as they upgrade Portage to 2.3.77 or -- if they upgraded
> already -- as their caches expire (24hrs).
> 
> The new layout is mostly a bow towards mirror admins, for some of whom
> having a 6+ files in a single directory have been a problem. 
> However, I suppose some of you also found e.g. the directory index
> hardly usable due to its size.
This sounds like a filesystem issue. Do we know which filesystems are suffering?

ZFS should be fine. I believe ext2/ext3 have problems with this many files. 
ext4 is probably okay, but don’t quote me on that.
> 
> Throughout a transitional period (whose exact length hasn't been decided
> yet), both layouts will be available.  Afterwards, the old layout will
> be removed from mirrors.  This has a few implications:
> 
> 1. Users who don't upgrade their package managers in time will lose
> the ability of fetching from Gentoo mirrors.  This shouldn't be that
> much of a problem given that the core software needed to upgrade Portage
> should all have reliable upstream SRC_URIs.
> 
> 2. mirror://gentoo/file URIs will stop working.  While technically you
> could use mirror://gentoo/XX/file, I'd rather recommend finally
> discarding its usage and moving distfiles to devspace.
> 
> 3. Directly fetching files from distfiles.gentoo.org will become
> a little harder.  To fetch a distfile named 'foo-1.tar.gz', you'd have
> to use something like:
> 
> $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2
> 1b
> $ wget http://distfiles.gentoo.org/distfiles/1b/foo-1.tar.gz
> ...
> 
> 
> Alternatively, you can:
> 
> $ wget http://distfiles.gentoo.org/distfiles/INDEX
> 
> and grep for the right path there.  This INDEX is also a more
> lightweight alternative to HTML indexes generated by the servers.
> 
> 
> If you're interested in more background details and some plots, see [1].
> 
> [1] 
> https://dev.gentoo.org/~mgorny/articles/improving-distfile-mirror-structure.html
> 
> -- 
> Best regards,
> Michał Górny
> 

Re: [gentoo-dev] [RFC] Adding 'GPL-2-only', 'GPL-3-only' etc. license variants for better auditing

[Richard Yao]

> On Sep 21, 2019, at 12:09 PM, Michał Górny  wrote:
> 
> Hi,
> 
> TL;DR: I'd like to replace 'GPL-2' with 'GPL-2-only' etc., having
> the former trigger QA warning asking the dev to double-check if it's
> 'GPL-2-only' or 'GPL-2+'.
> 
> 
> GNU Licenses currently don't carry an upgrade clause -- instead, authors
> are expected to decide whether they permit upgrade to newer versions of
> the license in question, or require users to stick with their version of
> choice.
> 
> Their decision is normally indicated in copyright notices on top
> of source files.  Those that permit upgrade usually state 'either
> version N of the License, or (at your option) any later version.', while
> others remove the 'or...' or even replace with 'only' (sometimes
> removing 'either', sometimes leaving it ;-)).
> 
> The truth is, many developers don't go that far to verify it.  Instead,
> they usually look at 'COPYING' or 'LICENSE', read the version there
> and put 'GPL-2', 'GPL-3' etc. in the ebuild.  It doesn't help that
> GitHub does the same and shows the result as easy-to-read note on top of
> repo.
> 
> 
> For some time I've been reviewing packages I'm (co-)maintaining, as well
> as proxy-maint submissions for this particular problem.  However,
> surprisingly many projects actually go the 'version N only' route, even
> in middle of environments that are 'N+' like Xfce.  As a result, I've
> ended up rechecking the same packages over and over again to the point
> of starting to add comments saying 'yes, this is GPL-2 only'.
> 
> I'd like to propose to employ a more systematic method of resolving this
> problem.  I would like to add additional explicit 'GPL-n-only' licenses,
> and discourage using short 'GPL-n' in favor of them.  The end result
> would be three licenses per every version/variant, e.g.:
> 
>  GPL-2-only -- version 2 only
>  GPL-2+ -- version 2 or newer
>  GPL-2  -- might be either, audit necessary
> 
> The main idea is that we'd be able to easily find 'non-audited' packages
> with GPL-2 entries, and replace them with either GPL-2+ or GPL-2-only
> after auditing.  While technically it would still be possible for people
> to wrongly set LICENSE to GPL-2-only, I think this explicit distinction
> will help people notice that there actually is a deeper difference,
> and it will still catch people who just type 'GPL-n' without looking
> into the license directory.
My read of this and the comments is that it boils down to getting people to do 
the right thing and ensuring that they did. If anyone does not already 
understand this, we need to have a talk with them about it.

Also, for things like the Linux kernel where some files lack the or later 
version clause, this is going to end up with us doing GPL-2-only and GPL-2+ at 
the same time. Is this really what we want to do there?
> 
> 
> For a start, I'd only go for adding the '-only' variants to the most
> common licenses, i.e. GPL-2, -3, LGPL-2, -2.1, -3, AGPL-3, maybe some
> FDL versions.  I don't think we need this for the long 'exception'
> variants -- I suspect that if someone did research enough to notice
> the exception, then most likely he would also notice the 'or newer'.
> 
> 
> WDYT?
> 
> -- 
> Best regards,
> Michał Górny
> 

Re: [gentoo-dev] Killing ~x86-fbsd?

[Richard Yao]

> On Jan 27, 2019, at 1:58 PM, Michał Górny  wrote:
> 
> Hello,
> 
> I'm going to keep this short.  Gentoo/FreeBSD is on life support for
> quite some time already.  The little work happening around it is all
> focused on amd64.  I'm not aware of any developer running ~x86-fbsd
> at this point, and all keywording requests on this arch aren't met with
> response.
> 
> The single profile is marked exp and the depgraph is continuously
> degrading.  At this point, it would probably be simpler to start over
> (and possibly copy keywords from ~amd64-fbsd) than to try to fix
> the current state.
> 
> Therefore, I'd like to propose that we remove the profiles and kill
> the arch altogether.  Is anyone opposed to that?
This will probably make life easier for people. I don’t see a problem with it.
> 
> -- 
> Best regards,
> Michał Górny

Re: [gentoo-dev] [RFC] C++ standard in ebuilds

[Richard Yao]

> On Sep 17, 2018, at 12:40 PM, Vadim A. Misbakh-Soloviov  
> wrote:
> 
> I'd prefer to wait another replies on the list for the main theme of this e-
> mail, but this problem also affects C (so, as **c**flags and C standards), so 
> solution shoudn't be c++ specific, imho.
You would think that, but the C standard version does not effect ABI 
compatibility. We had a very long discussion about this in OpenZFS because the 
other platforms were using C99 while we had backported everything to C89 on 
Linux because of the Linux kernel. No one could find a single example of ABI 
breakage caused by mixing and matching C89 and C99 (C99 LKM and C89 kernel). 
After a few years of not a single example being raised, the Linux team lead 
opted to adopt C99.

Now, there are flags affecting the ABI, but those are separate from the C 
standard version. As for C++, mixing standard versions does go badly because 
new language features require ABI changes.
> 
> 
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 14, 2018, at 7:07 PM, Sergei Trofimovich  wrote:
> 
> On Fri, 14 Sep 2018 11:54:57 -0400
> Richard Yao  wrote:
> 
>>>> My read of this is that the warning occurs regardless of optimization 
>>>> level, but it could somehow be improved by optimization.
>>>> 
>>>> As for the last, it is for uninitialized variable reads. However, I think 
>>>> you are misinterpreting the claim. The way that optimization level could 
>>>> affect warning generation would be if the warning generation came after 
>>>> optimization passes that could hide reads. That means that -O3 would 
>>>> prevent the warning.
> ...
>> Either provide code examples that generate warnings in a way that 
>> demonstrates that I am incorrect
> 
> To make code behave differently it needs substantial amount of code
> to provide you an example. You need to him O2<->O3 behaviour delta
> after all. But I will try (for a different warning, it should not matter
> much).
Thanks. I had been incorrect about -O3 giving not us some additional 
information for warnings. My apologies for the confusion.
> 
> Below is a reduced example of a larger C++ program.
> Many thanks to Ulya for providing recent example!
> 
> In this example -O3 manages to inline/const-propagate deep enough
> and find out potential null-deref. -O2 was not able to do it.
> 
> $ bash -x ./mk_.sh
> + LANG=C
> + g++-8.2.0 -O2 -c 1.cc -Wnull-dereference
> + g++-8.2.0 -O3 -c 1.cc -Wnull-dereference
> 1.cc: In function 'bool foo(std::vector)':
> 1.cc:3:22: warning: null pointer dereference [-Wnull-dereference]
>   typename h = e - d >> *g;
>~~^
> 1.cc:3:22: warning: null pointer dereference [-Wnull-dereference]
>   typename h = e - d >> *g;
>~~^
> 
> $ cat 1.cc
> #include 
> template  a c(a d, a e, b f, int *g) {
>  typename h = e - d >> *g;
>  for (; h;) if (f(*d)) if (f(*d)) return d;
>  return d;
> }
> template  i o(i d, i e, b f, int *g) {
>  return c(d, e, f, g);
> }
> template  i oo(i d, i e, b f, int *g) {
>  return c(d, e, f, g);
> }
> template  j k(j d, j e, b f, int *g) {
>  return o(d, e, f, g);
> }
> template  j kk(j d, j e, b f, int *g) {
>  return oo(d, e, f, g);
> }
> bool cmpp(int);
> bool foo(std::vector l) {
>  std::vector::const_iterator ib,
>  ie = l.end(), m, n = k(ib, ie, cmpp, 0) = kk(m, ie, cmpp, 0);
>  return m == n;
> }
> 
> -- 
> 
>  Sergei
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 14, 2018, at 5:28 PM, Fabian Groffen  wrote:
> 
> On 15-09-2018 00:07:12 +0300, Alon Bar-Lev wrote:
>>> 
>>> Perhaps, if one persists on going this route, only do this for platforms
>>> that upstream supports, such that arches which will suffer from this
>>> (typically ppc, sparc, ...) don't have to be blocked by this.
>> 
>> Exactly in these cases the -Werror is useful as if upstream expects no
>> warnings then any warning should block installation and trigger bug
>> report. In Gentoo in many cases we use packages on platform has no
>> access to, our feedback to upstream is valuable. A great example is
>> gnutls in which we collectively (maintainer, unstable users,
>> architecture teams, stable users) found issues on architectures that
>> almost nobody other than Gentoo has access to.
>> 
> 
> I don't believe Gentoo users are (supposed to be) an extension of
> upstreams.  If upstreams insist on that, they should make their software
> non-free, adding a non-modification clause or something.  In any case,
> it is not Gentoo's job IMHO.  In the end it is Gentoo who needs to care
> for its users.  I prefer we do that by giving them an option to become
> that extension of upstream, e.g. by USE=upstream-cflags, which Gentoo
> disables by default.
I am in complete agreement on this. Users should not be guinea pigs to help 
upstream unless they opt into it.
> 
> As maintainer and/or enthusiastic user, like you wrote for gnutls, I
> would be more than happy to provide build logs/errors for all the arches
> I have access to.  So like I wrote before, I think we should consider
> case-by-case basis to make it easy to do so.
> 
> Fabian
> 
> -- 
> Fabian Groffen
> Gentoo on a different level

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 14, 2018, at 4:20 PM, Michael Orlitzky  wrote:
> 
> On 09/14/2018 03:58 PM, Richard Yao wrote:
>>> 
>>> No one has answered the question: what do you do when a stable package
>>> breaks because of a new warning?
>>> 
>>> ...>
>> Wouldn’t this be largely covered as part of GCC stabilization? We could 
>> reserve the right to kill -Werror in a package where it blocks GCC 
>> stabilization if the maintainer does not handle it in a timely manner.
>>> 
> 
> They would be uncovered during GCC stabilization, but then you're right
> back in the original situation: how do you fix the stable package? The
> only answer that doesn't violate some other policy is to patch it in a
> new revision and wait for it to stabilize again.
This depends on the issue.
> 
> Other questions arise: Do we block stabilization of clang et al.?
We probably should start doing that once Clang is able to build everything, but 
someone would need to volunteer to handle it. It is a big job.
> 
> If we can simply remove -Werror because it's been a month, were the
> warnings ever really important to begin with?
That was a suggestion to handle maintainer non-response. You can already do 
whatever you want if the maintainer is non-responsive after telling him in a 
bug that you will do something if a response is not made in a reasonable period 
(e.g. 2 weeks). I am just pointing it out.
> 
> How many packages do we want to make the toolchain team stop and fix
> before they can do their jobs?
Presumably, the maintainers would handle this. If they cannot, they should not 
be honoring upstream’s -Werror policy.

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 14, 2018, at 3:29 PM, Michael Orlitzky  wrote:
> 
>> On 09/14/2018 01:52 PM, Rich Freeman wrote:
>> 
>> Wouldn't the flip side of this be demonstrating that this has actually
>> caused issues?  If following upstream discovers no bugs and also
>> causes no issues, why not leave it to maintainer discretion?
>> 
> 
> We know it causes issues, there are hundreds of bugs about it (bugzilla
> stops counting at 500 on a search for "Werror").
> 
> No one has answered the question: what do you do when a stable package
> breaks because of a new warning?
> 
> If there's no answer to that question that doesn't involve making an
> unofficial in-place downstream-only edit to a piece of code that is (by
> the opposing argument) intensely security-critical in a stable package,
> then we're all wasting our time talking about this.
Wouldn’t this be largely covered as part of GCC stabilization? We could reserve 
the right to kill -Werror in a package where it blocks GCC stabilization if the 
maintainer does not handle it in a timely manner.
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

On 09/13/2018 07:36 AM, Richard Yao wrote:
> 
> 
>> On Sep 12, 2018, at 6:55 PM, Thomas Deutschmann  wrote:
>>
>>> On 2018-09-12 16:50, Rich Freeman wrote:
>>> There is also the case where we want these warnings to block
>>> installation, because the risk of there being a problem is too great.
>>
>> I really disagree with that. So many devs have already said multiple
>> times in this thread that "-Werror" is only turning existing warnings
>> into fatal errors but "-Werror" itself doesn't add any new checks and
>> more often requires "-O3" to be useful.
> The way that compilers work is that the warnings are generated in the front 
> end while the optimization level affects the backend. That means that -O3 has 
> no effect on the code that does error generation. This remark about -O3 being 
> needed to make -Werror useful is just plain wrong. 
I had a discussion off list about this. My remarks on this were
incorrect. However, I strongly disagree that -O3 is necessary for
diagnostics to be useful.
>> So let's turn this around: Please show us a *real* case within Gentoo
>> where "-Werror" prevented a real problem which wouldn't otherwise being
>> noticed. E.g. show us a package which was merged on user's system,
>> replacing a working previous version of that package causing *real*
>> problems which could have been prevented if package would have set
>> "-Werror".
>>
>> Unless you can do that we don't really need to discuss this. Simply
>> because everyone interested in "-Werror" *can* set that flag via CFLAGS,
>> even just per package, whereas the majority, not interested in this,
>> cannot do the same to filter "-Werror". Nobody advocating for "-Werror"
>> replied to that fact yet.
>>
>>
>> -- 
>> Regards,
>> Thomas Deutschmann / Gentoo Linux Developer
>> C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
>>
> 
> 




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

On 09/13/2018 12:03 PM, Fabian Groffen wrote:
> On 13-09-2018 07:36:09 -0400, Richard Yao wrote:
>>
>>
>>> On Sep 12, 2018, at 6:55 PM, Thomas Deutschmann  wrote:
>>>
>>>> On 2018-09-12 16:50, Rich Freeman wrote:
>>>> There is also the case where we want these warnings to block
>>>> installation, because the risk of there being a problem is too great.
>>>
>>> I really disagree with that. So many devs have already said multiple
>>> times in this thread that "-Werror" is only turning existing warnings
>>> into fatal errors but "-Werror" itself doesn't add any new checks and
>>> more often requires "-O3" to be useful.
>> The way that compilers work is that the warnings are generated in the front 
>> end while the optimization level affects the backend. That means that -O3 
>> has no effect on the code that does error generation. This remark about -O3 
>> being needed to make -Werror useful is just plain wrong. 
> 
> Huh?  -O3 enables more checks, which can generate more warnings.  -O3
> isn't "needed", but if upstream is so interested in clean and correct
> code, they should've fixed all warnings in the first place and thus
> enabled all of them.

That wasn't how I read this:

> Also, consider that for -Werror to be "better", you also need -O3 in
order to activate the "proper" compiler checks like "variable set but
never used" ones.

But I'll accept that I misunderstood.

> In fact, I expect every sane upstream to use "-O3
> -Wall -Werror" in one of their automated builds.  Not that this catches
> anything useful on x86{,_64} when there is for instance use of signed
> and unsigned char types, so it isn't conclusive.
> 
> The whole point in here is that -Werror doesn't add much if you care.
> The whole point why it is not desired in Gentoo is that users don't
> necessarily are developers, or even interested in fixing warnings --
> regardless whether they point to real problems or not.
> 
> If there are real problems in a package (exposed by a compiler or not)
> then this should ideally stand out during ~arch testing, or even before
> when the Gentoo maintainer examines the build (might even use -Werror
> for his own purposes).  If such code ends up in stable arch we just made
> a stabilisation mistake, or got royally messed up by upstream, depending
> how you look at it.
> 
> Fabian
> 




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

On 09/14/2018 12:40 PM, Alon Bar-Lev wrote:
> On Fri, Sep 14, 2018 at 12:34 AM Sergei Trofimovich  wrote:
>>
>> On Tue, 11 Sep 2018 12:44:38 +0300
>> Alon Bar-Lev  wrote:
>>
>> I'm personally in favour of not allowing -Werror
>> to be in build system unconditionally.
>>
>> Maintainer is free to implement --enable-werror any way
>> it's convenient to run on their own extended sanity checks
>> and optionally expose it to users. Be it USE flag or
>> EXTRA_ECONF option.
> 
> This discussion is not for downstream to have a more strict policy
> than upstream. People try to hijack discussion and introduce noise to
> de-focus the discussion.
> 
> Downstream policy cannot be more strict than upstream as then every
> change upstream is doing downstream need to rebase and invest in even
> more changes.
> 
> This discussion is to follow upstream strict policy if upstream proves
> that it stands behind it and downstream is willing to follow.
I don't think we should do that unless we provide a USE flag for users
to opt into the behavior. Forcing it on users is problematic for the
reasons others stated. However, letting them opt into the behavior is
reasonable.

In the case of sys-fs/zfs, enabling -Werror (which includes -Wall) on
USE=debug is following upstream's wishes to build debug builds with -Werror.
> 
> For your question: No. Downstream should not add -Werror to upstream
> package, not in a parameter or USE flag, as this will probably break
> and cause a queue of downstream patches.
> 
>>> I would like to suggest a modify our policy with the following
>>> exception clause: Package maintainer may decide to keep upstream
>>> -Werror as long as he is willing to resolve all issues resulting from
>>> -Werror as if it was a blocker bug.
>>
>> Do you intend to keep -Werror for case when ebuild goes
>> stable as well?
> 
> Correct.
> 
>> If you do it then what is your workflow to fix breakages
>> appearing in stable packages caused by minor environment
>> changes like toolchain tweaks?
> 
> Correct.
> 
>> Add another round of stabilization on each arch team? It
>> sounds like your default assumption that code needs to be
>> changed in a semantically significant way: add annotations
>> that might not like some toolchains, remove unused code.
> 
> No dependency of toolchain nor annotations.
> A strict policy of no warnings will require changes as dependencies
> including toolchain are progressing.
> This creates an overhead for selected packages.
> A maintainer and the non-stable team should be able to know the package 
> status.
> Preferably this may be done by automation, I appreciate the work of
> Toralf Förster with tinderbox to automate check various cases.
> When a new slot of toolchain is available, maintainers should check
> their packages in any case, there are other issues and side affects
> that we experienced, especially in C++ packages.
> For these packages usually there are 3 for each slot: the current
> stable, the next stable and the non-stable, so the overhead is
> constrained.
> 
>> Or patch package in-place without bumping? None of options
>> sound optimal compared to dropping -Werror.
> 
> Success of build is not the only concern although I see people here
> that are interested only in that.
> Patching upstream package and/or change upstream quality policy is
> something that we should avoid as well to maintain upstream warranty.
> 
>>> The package maintainer decision may be based on the package state and
>>> the relationship with upstream, but basically, it is his decision as
>>> long as issues are fixed in timely manner, it is his overhead after
>>> all.
>>
>> I agree that maintainer's will and upstream's will should be
>> respected and it's not something needs to be absolutely
>> enforced all the time.
>>
>> Personally I disagree -Werror on end-user machine is worth
>> it (or cppcheck, or coverity round-trip run is worth running
>> on user's machine unconditionally).
>>
>> Unused variable is a good example of typical benign warning:
>> it was there all the time, it's not a new bug and does not
>> warrant need for an immediate fix.
> 
> Unused variable is a good example of CRITICAL potential issue, the bug
> that triggered this this discussion has a return code that was not
> used. The permutation was not tested by upstream as it rarely used, it
> was not tested by me either by the same reason, both is a mistake.
> Fortunately, someone else tested this permutation and his build
> failed, triggered a bug. If -Werror has not been used we would not
> have known about this issue. In many cases these happen in
> architecture that maintainer nor upstream have access to. In this
> specific case I went over the code history to the time the return code
> have been used and determined that this indeed should be ignored,
> imagine the opposite. A patch was submitted to upstream to confirm
> resolution as any issue in code, upstream confirmed and merged this in
> timely fashion. Bottom line we all (Gentoo, 

[gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 13, 2018, at 8:54 PM, Georg Rudoy <0xd34df...@gmail.com> wrote:
> 
>> On 14.09.2018 at 0:44 user Richard Yao  wrote:
>> This is a really odd design decision by the GCC developers. With other 
>> compilers, the separation between front end and backend is strong enough 
>> that you will never have this sort of thing. It does not seem necessary to 
>> me either. :/
I didn't want to get into this, but GCC's backend is actually split into
a middle-end, that does architecture independent optimizations and a
backend, that generates assembly code (and likely does architecture
specific optimization):

https://en.wikibooks.org/wiki/GNU_C_Compiler_Internals/GNU_C_Compiler_Architecture

You can get it to dump what appears to be the input to the middle end by
doing -fdump-translation-unit. You can get GCC to dump each stage of the
middle end using -fdump-tree-all. Presumably, any warnings that depend
on optimization won't be generated, although someone would need to study
the code to verify that.

I suspect quite a few people talking about what -Werror -Wall does have
never actually touched the internals of a compiler and perhaps that
should change. I won't claim to be an expert, but I have minor
experience with front-end development, writing code to do error
generation, etcetera. Honestly, when I first looked at GCC's sources, I
decided that I would be happier never looking at them ever again, but I
am starting to reconsider that decision.
> 
> You might be able to perform certain additional data/control flow analysis 
> after things like inlining, dead code removal or devirtualization.
Do you have any examples? I am having some trouble making a test case
that shows a different in behavior at different optimization levels
either way.

Here is an example:

__attribute__((always_inline))
static inline int
test(int x) {
return x;
}

int main() {
int x;

return (test(x)*0);
}

GCC 7.3.0 will emit a warning regardless of optimization level provided
that -Wall is passed. However, it will not emit a warning for this at
any optimization level:

int main() {
int x;

return (x*0);
}

> 
> Moving that logic to the frontend would require essentially duplicating 
> what's the optimizer's gonna do anyway, which might have negative effects on 
> compilation times (both with and without optimizations) and compiler code 
> maintenance.
If it is easier to use the optimizer's infrastructure to figure this
out, then the code could be written to call it to do analysis as a
pseudo-optimization pass, which GCC actually appears to do when it runs
its "Function test" "optimization" pass. There is no code maintenance
burden.

Fabian was right about needing -Wall for -Werror to catch many things
because most warnings are off by default. However, I am extremely
skeptical that the optimization level has significant error. -Wall gives
us whatever we need and so far, I cannot find a code example showing
differences in warning generation at different optimization levels. I'll
accept the idea that the warning quality might change (although I don't
fully understand the reasoning for it) by the GCC documentation.

However, I find the idea that -O3 will make a warning appear that didn't
otherwise appear to be very difficult to accept. The documentation
claims that optimization passes can effect emission of warnings for
uninitialized variables, but the only way that I could imagine that
happening would be if an optimization pass made x*0 into 0 before the
code generating the warning is run. This is the complete opposite of
what is being claimed here. In some actual tests, I am unable to get GCC
to emit a warning for that at any optimization level. I am also unable
to get it to behave differently for uninitialized variables at different
optimization levels no matter what idea I try.
> 
> 
> 
> -- 
>  Georg Rudoy
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 13, 2018, at 11:35 PM, Matt Turner  wrote:
> 
> On Thu, Sep 13, 2018 at 5:44 PM Richard Yao  wrote:
>>> On Sep 13, 2018, at 7:21 PM, Matt Turner  wrote:
>>> 
>>> On Thu, Sep 13, 2018 at 4:13 PM Richard Yao  wrote:
>>>>> On Sep 13, 2018, at 12:03 PM, Fabian Groffen  wrote:
>>>>> 
>>>>>> On 13-09-2018 07:36:09 -0400, Richard Yao wrote:
>>>>>> 
>>>>>> 
>>>>>>>> On Sep 12, 2018, at 6:55 PM, Thomas Deutschmann  
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> On 2018-09-12 16:50, Rich Freeman wrote:
>>>>>>>> There is also the case where we want these warnings to block
>>>>>>>> installation, because the risk of there being a problem is too great.
>>>>>>> 
>>>>>>> I really disagree with that. So many devs have already said multiple
>>>>>>> times in this thread that "-Werror" is only turning existing warnings
>>>>>>> into fatal errors but "-Werror" itself doesn't add any new checks and
>>>>>>> more often requires "-O3" to be useful.
>>>>>> The way that compilers work is that the warnings are generated in the 
>>>>>> front end while the optimization level affects the backend. That means 
>>>>>> that -O3 has no effect on the code that does error generation. This 
>>>>>> remark about -O3 being needed to make -Werror useful is just plain wrong.
>>>>> 
>>>>> Huh?  -O3 enables more checks, which can generate more warnings.
>>>> 
>>>> What checks are those? -O3 affects backend optimization while warnings are 
>>>> generated by the front end. Once the immediate representation is 
>>>> generated, there are no other warnings aside from those from the linker.
>>> 
>>> https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
>>> 
>>> Search for "depend on"
>>> 
>>> -> [...] estimated based on heuristics that depend on thelevel
>>> argument and on optimization
>>> 
>>> -> Because these warnings depend on optimization [...]
>>> 
>>> Yes, warnings are dependent on optimization level.
>> 
>> There are three such options. The first two are for format statements:
>> 
>>> “When the exact number of bytes written by a format directive cannot be 
>>> determined at compile-time it is estimated based on heuristics that depend 
>>> on the level argument and on optimization. While enabling optimization will 
>>> in most cases improve the accuracy of the warning, it may also result in 
>>> false positives. “
>> 
>> My read of this is that the warning occurs regardless of optimization level, 
>> but it could somehow be improved by optimization.
>> 
>> As for the last, it is for uninitialized variable reads. However, I think 
>> you are misinterpreting the claim. The way that optimization level could 
>> affect warning generation would be if the warning generation came after 
>> optimization passes that could hide reads. That means that -O3 would prevent 
>> the warning.
>> 
>> This is a really odd design decision by the GCC developers. With other 
>> compilers, the separation between front end and backend is strong enough 
>> that you will never have this sort of thing. It does not seem necessary to 
>> me either. :/
> 
> I'm sorry, but you really don't know what you're talking about.

Either provide code examples that generate warnings in a way that demonstrates 
that I am incorrect or withdraw that ad hominem.

> I've
> already told you once that you were just adding noise to this
> conversation.
I can understand the remarks about USE=debug being considered noise given that 
people seem to think that -Werror is a boolean thing where it is either on for 
any combination of USE flags or off for any combination of USE flags. However, 
discussing how compiler internals impacts how -Werror works is definitely not 
noise.

You cannot have a discussion about how -Werror should be handled without 
understanding what I actually does.

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 13, 2018, at 7:21 PM, Matt Turner  wrote:
> 
> On Thu, Sep 13, 2018 at 4:13 PM Richard Yao  wrote:
>>> On Sep 13, 2018, at 12:03 PM, Fabian Groffen  wrote:
>>> 
>>>> On 13-09-2018 07:36:09 -0400, Richard Yao wrote:
>>>> 
>>>> 
>>>>>> On Sep 12, 2018, at 6:55 PM, Thomas Deutschmann  
>>>>>> wrote:
>>>>>> 
>>>>>> On 2018-09-12 16:50, Rich Freeman wrote:
>>>>>> There is also the case where we want these warnings to block
>>>>>> installation, because the risk of there being a problem is too great.
>>>>> 
>>>>> I really disagree with that. So many devs have already said multiple
>>>>> times in this thread that "-Werror" is only turning existing warnings
>>>>> into fatal errors but "-Werror" itself doesn't add any new checks and
>>>>> more often requires "-O3" to be useful.
>>>> The way that compilers work is that the warnings are generated in the 
>>>> front end while the optimization level affects the backend. That means 
>>>> that -O3 has no effect on the code that does error generation. This remark 
>>>> about -O3 being needed to make -Werror useful is just plain wrong.
>>> 
>>> Huh?  -O3 enables more checks, which can generate more warnings.
>> 
>> What checks are those? -O3 affects backend optimization while warnings are 
>> generated by the front end. Once the immediate representation is generated, 
>> there are no other warnings aside from those from the linker.
> 
> https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
> 
> Search for "depend on"
> 
> -> [...] estimated based on heuristics that depend on thelevel
> argument and on optimization
> 
> -> Because these warnings depend on optimization [...]
> 
> Yes, warnings are dependent on optimization level.

There are three such options. The first two are for format statements:

> “When the exact number of bytes written by a format directive cannot be 
> determined at compile-time it is estimated based on heuristics that depend on 
> the level argument and on optimization. While enabling optimization will in 
> most cases improve the accuracy of the warning, it may also result in false 
> positives. “

My read of this is that the warning occurs regardless of optimization level, 
but it could somehow be improved by optimization.

As for the last, it is for uninitialized variable reads. However, I think you 
are misinterpreting the claim. The way that optimization level could affect 
warning generation would be if the warning generation came after optimization 
passes that could hide reads. That means that -O3 would prevent the warning.

This is a really odd design decision by the GCC developers. With other 
compilers, the separation between front end and backend is strong enough that 
you will never have this sort of thing. It does not seem necessary to me 
either. :/

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 13, 2018, at 12:03 PM, Fabian Groffen  wrote:
> 
>> On 13-09-2018 07:36:09 -0400, Richard Yao wrote:
>> 
>> 
>>>> On Sep 12, 2018, at 6:55 PM, Thomas Deutschmann  wrote:
>>>> 
>>>> On 2018-09-12 16:50, Rich Freeman wrote:
>>>> There is also the case where we want these warnings to block
>>>> installation, because the risk of there being a problem is too great.
>>> 
>>> I really disagree with that. So many devs have already said multiple
>>> times in this thread that "-Werror" is only turning existing warnings
>>> into fatal errors but "-Werror" itself doesn't add any new checks and
>>> more often requires "-O3" to be useful.
>> The way that compilers work is that the warnings are generated in the front 
>> end while the optimization level affects the backend. That means that -O3 
>> has no effect on the code that does error generation. This remark about -O3 
>> being needed to make -Werror useful is just plain wrong. 
> 
> Huh?  -O3 enables more checks, which can generate more warnings.

What checks are those? -O3 affects backend optimization while warnings are 
generated by the front end. Once the immediate representation is generated, 
there are no other warnings aside from those from the linker.
>  -O3
> isn't "needed", but if upstream is so interested in clean and correct
> code, they should've fixed all warnings in the first place and thus
> enabled all of them.  In fact, I expect every sane upstream to use "-O3
> -Wall -Werror" in one of their automated builds.  Not that this catches
> anything useful on x86{,_64} when there is for instance use of signed
> and unsigned char types, so it isn't conclusive.
> 
> The whole point in here is that -Werror doesn't add much if you care.
> The whole point why it is not desired in Gentoo is that users don't
> necessarily are developers, or even interested in fixing warnings --
> regardless whether they point to real problems or not.
> 
> If there are real problems in a package (exposed by a compiler or not)
> then this should ideally stand out during ~arch testing, or even before
> when the Gentoo maintainer examines the build (might even use -Werror
> for his own purposes).  If such code ends up in stable arch we just made
> a stabilisation mistake, or got royally messed up by upstream, depending
> how you look at it.
> 
> Fabian
> 
> -- 
> Fabian Groffen
> Gentoo on a different level

Re: [gentoo-dev] acceptable alternatives to -Werror, was: Changing policy about -Werror

[Richard Yao]

> On Sep 12, 2018, at 8:23 PM, Chí-Thanh Christopher Nguyễn 
>  wrote:
> 
> Rich Freeman schrieb:
>>> Requirements:
>>> 
>>> * Do not fail to build/install when a warning is encountered
>> On a particularly critical package like a filesystem, wouldn't we want
>> to still fail to install when a warning is encountered?
> 
> Installation will proceed, but the user will get a big fat warning that the 
> sys-fs/zfs package is potentially broken.

The way that it works is that a user explicitly must enable USE=debug for 
-Werror to apply. The package itself’s use of -Werror in USE=debug does not 
prevent corruption issues so much as it detects them because the kernel code is 
built in userspace. sys-fs/zfs-kmod is what matters for this. USE=debug does 
not turn on -Werror there because we don’t inject it into the kernel build 
system. If such a bug is detected and a user proceeds to use the rebuilt kernel 
module, things could go poorly.

Anyway, I seem to have injected noise into the conversation by bringing this up 
because the discussion is whether -Werror by default should be allowed. I only 
commented because I felt that it should be an exception in the case of 
USE=debug or an explicit USE flag for Werror (e.g. USE=Werror).
> 
>> I get that users might quit if packages don't install, but I'm not
>> sure that a filesystem corruption is going to make them any happier...
> 
> If the user recognizes this as a critical package, then they can do the 
> research before deciding on whether to use the package as is, attempt to 
> downgrade, or wait until a fix is released.
> 
> 
> Best regards,
> Chí-Thanh Christopher Nguyễn
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 12, 2018, at 6:55 PM, Thomas Deutschmann  wrote:
> 
>> On 2018-09-12 16:50, Rich Freeman wrote:
>> There is also the case where we want these warnings to block
>> installation, because the risk of there being a problem is too great.
> 
> I really disagree with that. So many devs have already said multiple
> times in this thread that "-Werror" is only turning existing warnings
> into fatal errors but "-Werror" itself doesn't add any new checks and
> more often requires "-O3" to be useful.
The way that compilers work is that the warnings are generated in the front end 
while the optimization level affects the backend. That means that -O3 has no 
effect on the code that does error generation. This remark about -O3 being 
needed to make -Werror useful is just plain wrong. 
> So let's turn this around: Please show us a *real* case within Gentoo
> where "-Werror" prevented a real problem which wouldn't otherwise being
> noticed. E.g. show us a package which was merged on user's system,
> replacing a working previous version of that package causing *real*
> problems which could have been prevented if package would have set
> "-Werror".
> 
> Unless you can do that we don't really need to discuss this. Simply
> because everyone interested in "-Werror" *can* set that flag via CFLAGS,
> even just per package, whereas the majority, not interested in this,
> cannot do the same to filter "-Werror". Nobody advocating for "-Werror"
> replied to that fact yet.
> 
> 
> -- 
> Regards,
> Thomas Deutschmann / Gentoo Linux Developer
> C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

On Sep 12, 2018, at 4:28 PM, Andreas K. Huettel  wrote:

>> If a package really ought to have
>> -Werror due to a very good reason and is properly maintained to support it,
>> then there is nothing wrong with inventing a USE flag to give users the
>> option of enforcing that.
> 
> There is something very *much* wrong with that.
> 
> 1) It's trivial to enforce -Werror via the packages.env mechanism on specific 
> packages (e.g. those that you maintain).

That really does not help the users see which packages explicitly support 
-Werror if they want it.
> 
> 2) Compared to that, an additional useflag introduces a lot of unnecessary 
> overhead at the package manager level *and* requires yet another level of 
> policies and Gentoo-specific definitions.
It occurs to me that this is really a debug feature, so it really ought to be 
turned on for USE=debug. Use of USE=debug in production is largely discouraged, 
so this could be fine. However, this is very much a case by case thing.
> 
> -- 
> Andreas K. Hüttel
> dilfri...@gentoo.org
> Gentoo Linux developer 
> (council, toolchain, perl, libreoffice, comrel)
> 
> 
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 10, 2018, at 5:48 PM, Richard Yao  wrote:
> 
> 
> 
>>> On Sep 10, 2018, at 5:27 PM, Kristian Fiskerstrand  wrote:
>>> 
>>>> On 9/10/18 11:21 PM, Kristian Fiskerstrand wrote:
>>>> On 9/10/18 11:19 PM, Chí-Thanh Christopher Nguyễn wrote:
>>>> It is indeed an insurmountable task to write code that is warning-free
>>>> from the beginning across architectures, compiler versions, etc. But
>>>> that is not the goal anyway. It is examining the situation and taking
>>>> appropriate action, and then applying a change to no longer cause that
>>>> particular warning (or make it non-fatal if the warning is bogus/harmless).
>>> 
>>> sure, but for upstreams that make this an explicit goal, do we really
>>> want to apply additional downstream pataches with the additional
>>> complexity that carries for build system (autotools re-generation that
>>> might make it unsupported upstream etc) ?
>>> 
>> 
>> in all fairness, for one of my upstream packages, SKS, we make -Werror
>> part of non-release versions but remove it for releases.
> This has been what sys-fs/zfs has been doing for years. The USE=-debug builds 
> get -Werror while USE=-debug builds omit it. I think this is probably the 
> solution here. USE=debug is meant to help catch bugs, even if some reports 
> might be false positives. What it means varies on a per-package basis. I 
> would call catching a security issue helping to catch bugs.
There is a typo here. USE=+debug builds get -Werror while USE=-debug omit 
-Werror.
> 
>> But there are
>> certain crypto related packages where you want the ensure it is properly
>> handle altogether, in particular where RNG is concerned as there isn't
>> really a proper way to test for it afterwards.. for other packages the
>> test suite is of great importance.. if the tests are proper there isn't
>> a great need, but sadly packages today doesn't really come with proper
>> test suits
>> 
>> -- 
>> Kristian Fiskerstrand
>> OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
>> fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
>> 
> 
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 10, 2018, at 5:27 PM, Kristian Fiskerstrand  wrote:
> 
>> On 9/10/18 11:21 PM, Kristian Fiskerstrand wrote:
>>> On 9/10/18 11:19 PM, Chí-Thanh Christopher Nguyễn wrote:
>>> It is indeed an insurmountable task to write code that is warning-free
>>> from the beginning across architectures, compiler versions, etc. But
>>> that is not the goal anyway. It is examining the situation and taking
>>> appropriate action, and then applying a change to no longer cause that
>>> particular warning (or make it non-fatal if the warning is bogus/harmless).
>> 
>> sure, but for upstreams that make this an explicit goal, do we really
>> want to apply additional downstream pataches with the additional
>> complexity that carries for build system (autotools re-generation that
>> might make it unsupported upstream etc) ?
>> 
> 
> in all fairness, for one of my upstream packages, SKS, we make -Werror
> part of non-release versions but remove it for releases.
This has been what sys-fs/zfs has been doing for years. The USE=-debug builds 
get -Werror while USE=-debug builds omit it. I think this is probably the 
solution here. USE=debug is meant to help catch bugs, even if some reports 
might be false positives. What it means varies on a per-package basis. I would 
call catching a security issue helping to catch bugs.
> But there are
> certain crypto related packages where you want the ensure it is properly
> handle altogether, in particular where RNG is concerned as there isn't
> really a proper way to test for it afterwards.. for other packages the
> test suite is of great importance.. if the tests are proper there isn't
> a great need, but sadly packages today doesn't really come with proper
> test suits
> 
> -- 
> Kristian Fiskerstrand
> OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
> fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 10, 2018, at 5:18 PM, Chí-Thanh Christopher Nguyễn 
>  wrote:
> 
> Fabian Groffen schrieb:
>>> On 09-09-2018 11:22:41 -0400, Richard Yao wrote:
>>> -Werror has caught bugs that could have resulted in data loss in ZFS in the 
>>> past thanks to it being built in userspace as part of zdb. So it is useful 
>>> for integrity too, not just security (although arguably, integrity is part 
>>> of security).
>> This is a misconception, as jer already pointed out.  Instead:
>> -Werror has forced you to take notice of problems that could have
>> resulted in data loss in ZFS ...
> 
> But in general it doesn't say when or how the problems became acute. Assuming 
> that the warning isn't bogus, it could have been the code relying on 
> undefined behavior, and the compiler changing the semantics of such behavior, 
> and introducing an accompanying warning at the same time.
This is precisely why every warning is considered a bug and -Werror is set when 
USE=debug is in use.
> 
> 
> Best regards,
> Chí-Thanh Christopher Nguyễn
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 10, 2018, at 4:59 PM, Mart Raudsepp  wrote:
> 
> Ühel kenal päeval, E, 10.09.2018 kell 22:56, kirjutas Kristian
> Fiskerstrand:
>>> On 9/10/18 10:51 PM, Matt Turner wrote:
>>> Consider again the bug that started this. The maintainer had not
>>> built
>>> this configuration. None of the arch teams had built this
>>> configuration until I did for the last architecture Cc'd. The patch
>>> committed doesn't change anything installed on the system, if not
>>> for
>>> Werror preventing the code from building.
>> 
>> one way to look at it though, is that it is a valuable upstream
>> contribution that this configuration produces the error, so Gentoo is
>> contributing to upstream development because of it.
> 
> And losing users and thus relevance in the process. Not everyone goes
> to bugzilla always and then waits for a fix, or fixes it themselves.
> Normal people wipe that stuff away and install an operating
> system/distribution that doesn't cause them grief in its place.

I would like to point out that if -Werror is set by say USE=debug, this is not 
an issue.

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 10, 2018, at 10:19 AM, Fabian Groffen  wrote:
> 
>> On 09-09-2018 11:22:41 -0400, Richard Yao wrote:
>> -Werror has caught bugs that could have resulted in data loss in ZFS in the 
>> past thanks to it being built in userspace as part of zdb. So it is useful 
>> for integrity too, not just security (although arguably, integrity is part 
>> of security).
> 
> This is a misconception, as jer already pointed out.  Instead:
> 
> -Werror has forced you to take notice of problems that could have
> resulted in data loss in ZFS ...
It did. That is why it is used as a debug feature only when USE=debug is set. 
USE=-debug does not use -Werror. USE=debug on that package is meant for people 
who want to help upstream catch bugs.
> 
> Also, consider that for -Werror to be "better", you also need -O3 in
> order to activate the "proper" compiler checks like "variable set but
> never used" ones.
I have had “set but never used” errors on -O2.
> 
>> Perhaps we could have another USE flag for -Werror where it is a security 
>> feature. e.g. USE=strict-compile-checks
> 
> You better run a static code analyser, such as the one you can hook up
> with Travis.  It usually points out real security problems such as
> races, which GCC doesn't do yet, as far as I'm aware.  Let alone
> trigger with -Werror.
We are using Coverity, but there is no one tool that catches all issues such 
that the compiler’s checks are redundant.
> 
> Fabian
> 
> 
> -- 
> Fabian Groffen
> Gentoo on a different level

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 9, 2018, at 1:09 PM, Richard Yao  wrote:
> 
> 
> 
>> On Sep 9, 2018, at 12:11 PM, Michał Górny  wrote:
>> 
>> On Sun, 2018-09-09 at 11:22 -0400, Richard Yao wrote:
>>>> On Sep 9, 2018, at 7:32 AM, Andrew Savchenko  wrote:
>>>> 
>>>> Hi!
>>>> 
>>>> Our current -Werror policy demands unconditional removal:
>>>> https://devmanual.gentoo.org/ebuild-writing/common-mistakes/index.html#-werror-compiler-flag-not-removed
>>>> 
>>>> I think this is wrong, see bugs 665464, 665538 for a recent
>>>> discussion why.
>>>> 
>>>> My point is that in *most* cases -Werror indeed should be removed,
>>>> because upstream rarely can keep up with all possible configure,
>>>> *FLAGS, compiler versions and arch combinations. But! In some cases
>>>> — especially for security oriented software — this flag may be
>>>> pertain and may be kept at maintainer's discretion.
>>>> 
>>>> The rationale is that -Werror usually points to dangerous
>>>> situations like uninitialized variables, pointer type mismatch or
>>>> implicit function declaration (and much more) which may lead to
>>>> serious security implications.
>>>> 
>>>> So, if maintainer has enough manpower to support this flag, we
>>>> should allow to keep it. Of course if it will cause long-standing
>>>> troubles (e.g. bugs opened for a long time) QA should have power to
>>>> remove it or demand its removal.
>>>> 
>>>> So my proposal is:
>>>> 
>>>> 1) Deprecate QA policy with unconditional demand of -Werror removal.
>>>> 2) Add to devmanual's chapter on -Werror an exception clause about
>>>> security-oriented software and maintainer's right to make final
>>>> decision.
>>> 
>>> -Werror has caught bugs that could have resulted in data loss in ZFS in the 
>>> past thanks to it being built in userspace as part of zdb. So it is useful 
>>> for integrity too, not just security (although arguably, integrity is part 
>>> of security).
>>> 
>>> Currently, sys-fs/zfs turns on -Werror when USE=debug is set. So far, 
>>> nobody has complained about USE=debug enforcing -Werror. USE=debug by 
>>> definition ought to be an exception.
>> 
>> Now that you know that you're violating a policy, please kindly fix
>> that.
> I already knew about the policy. However, USE=debug is by definition meant 
> for debug purposes. -Werror is helpful for debugging. There is nothing wrong 
> with turning it on for debugging. The normal builds don’t have USE=debug set 
> and -Werror is not used there.

By the way, if people insist on applying this policy to USE=debug, I am 
inclined to mask the USE flag. This would satisfy the policy, but I don’t think 
that is better than how things are now. Users already are warned not to set 
USE=debug globally and if they are setting it on a specific package, they are 
opting into the package’s definition of debug functionality.

I don’t see a problem with having -Werror as part of USE=debug. USE=debug on 
that package is meant to be an aid to upstream development and upstream in this 
case wants to know about any warnings.
>>> Perhaps we could have another USE flag for -Werror where it is a security 
>>> feature. e.g. USE=strict-compile-checks
>> 
>> Perhaps people could learn that Gentoo lets them alter CFLAGS, and stop
>> inventing USE flags for every flag the compiler supports.
>> 
>>>> 
>>>> Best regards,
>>>> Andrew Savchenko
>>> 
>>> 
>> 
>> -- 
>> Best regards,
>> Michał Górny
> 
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

On Sep 9, 2018, at 12:32 PM, Ulrich Mueller  wrote:

>> On Sun, 09 Sep 2018, Andrew Savchenko wrote:
> 
>> What I'm trying to do is to allow maintainers to keep -Werror if
>> they really want to do this, understand what they are doing and
>> have enough manpower to support this.
> 
> Bug 665464 has just proven that this doesn't work. That bug would not
> have happened if the policy had been followed. Also its fix (removal of
> an unused variable) should have been applied only upstream. I don't see
> a good reason for adding downstream patches that will make no difference
> for the resulting binary. At least not when the upstream package is
> maintained, and the issue will likely go away with one of the next
> releases.
> 
>> As can be seen from aforementioned bugs right now developer and
>> upstream support this to their best and yet QA team tries to
>> enforce -Werror drop using the brute force and ignoring active best
>> effort support. This should not happen.
> 
> See flameeyes's old blog post for the rationale why the current policy
> is in place:
> https://flameeyes.blog/2009/02/25/future-proof-your-code-dont-use-werror/
For every pointless check that fails -Werror, there is likely one that actually 
does matter. An unused variable could go either way if upstream intended to use 
that variable, but used another one by mistake (it happens).

Allowing users to opt into -Werror behavior on specific packages whose 
maintainers have a good reason to do it and are keeping up with things would be 
alright.
> 
> Ulrich
> 

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 9, 2018, at 12:11 PM, Michał Górny  wrote:
> 
> On Sun, 2018-09-09 at 11:22 -0400, Richard Yao wrote:
>>> On Sep 9, 2018, at 7:32 AM, Andrew Savchenko  wrote:
>>> 
>>> Hi!
>>> 
>>> Our current -Werror policy demands unconditional removal:
>>> https://devmanual.gentoo.org/ebuild-writing/common-mistakes/index.html#-werror-compiler-flag-not-removed
>>> 
>>> I think this is wrong, see bugs 665464, 665538 for a recent
>>> discussion why.
>>> 
>>> My point is that in *most* cases -Werror indeed should be removed,
>>> because upstream rarely can keep up with all possible configure,
>>> *FLAGS, compiler versions and arch combinations. But! In some cases
>>> — especially for security oriented software — this flag may be
>>> pertain and may be kept at maintainer's discretion.
>>> 
>>> The rationale is that -Werror usually points to dangerous
>>> situations like uninitialized variables, pointer type mismatch or
>>> implicit function declaration (and much more) which may lead to
>>> serious security implications.
>>> 
>>> So, if maintainer has enough manpower to support this flag, we
>>> should allow to keep it. Of course if it will cause long-standing
>>> troubles (e.g. bugs opened for a long time) QA should have power to
>>> remove it or demand its removal.
>>> 
>>> So my proposal is:
>>> 
>>> 1) Deprecate QA policy with unconditional demand of -Werror removal.
>>> 2) Add to devmanual's chapter on -Werror an exception clause about
>>> security-oriented software and maintainer's right to make final
>>> decision.
>> 
>> -Werror has caught bugs that could have resulted in data loss in ZFS in the 
>> past thanks to it being built in userspace as part of zdb. So it is useful 
>> for integrity too, not just security (although arguably, integrity is part 
>> of security).
>> 
>> Currently, sys-fs/zfs turns on -Werror when USE=debug is set. So far, nobody 
>> has complained about USE=debug enforcing -Werror. USE=debug by definition 
>> ought to be an exception.
> 
> Now that you know that you're violating a policy, please kindly fix
> that.
> 
>> Perhaps we could have another USE flag for -Werror where it is a security 
>> feature. e.g. USE=strict-compile-checks
> 
> Perhaps people could learn that Gentoo lets them alter CFLAGS, and stop
> inventing USE flags for every flag the compiler supports.

Do that and watch nearly everything break. If a package really ought to have 
-Werror due to a very good reason and is properly maintained to support it, 
then there is nothing wrong with inventing a USE flag to give users the option 
of enforcing that. It is better than letting users discover that via random 
trial and error. That just wastes people’s time.
> 
>>> 
>>> Best regards,
>>> Andrew Savchenko
>> 
>> 
> 
> -- 
> Best regards,
> Michał Górny

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 9, 2018, at 12:11 PM, Michał Górny  wrote:
> 
> On Sun, 2018-09-09 at 11:22 -0400, Richard Yao wrote:
>>> On Sep 9, 2018, at 7:32 AM, Andrew Savchenko  wrote:
>>> 
>>> Hi!
>>> 
>>> Our current -Werror policy demands unconditional removal:
>>> https://devmanual.gentoo.org/ebuild-writing/common-mistakes/index.html#-werror-compiler-flag-not-removed
>>> 
>>> I think this is wrong, see bugs 665464, 665538 for a recent
>>> discussion why.
>>> 
>>> My point is that in *most* cases -Werror indeed should be removed,
>>> because upstream rarely can keep up with all possible configure,
>>> *FLAGS, compiler versions and arch combinations. But! In some cases
>>> — especially for security oriented software — this flag may be
>>> pertain and may be kept at maintainer's discretion.
>>> 
>>> The rationale is that -Werror usually points to dangerous
>>> situations like uninitialized variables, pointer type mismatch or
>>> implicit function declaration (and much more) which may lead to
>>> serious security implications.
>>> 
>>> So, if maintainer has enough manpower to support this flag, we
>>> should allow to keep it. Of course if it will cause long-standing
>>> troubles (e.g. bugs opened for a long time) QA should have power to
>>> remove it or demand its removal.
>>> 
>>> So my proposal is:
>>> 
>>> 1) Deprecate QA policy with unconditional demand of -Werror removal.
>>> 2) Add to devmanual's chapter on -Werror an exception clause about
>>> security-oriented software and maintainer's right to make final
>>> decision.
>> 
>> -Werror has caught bugs that could have resulted in data loss in ZFS in the 
>> past thanks to it being built in userspace as part of zdb. So it is useful 
>> for integrity too, not just security (although arguably, integrity is part 
>> of security).
>> 
>> Currently, sys-fs/zfs turns on -Werror when USE=debug is set. So far, nobody 
>> has complained about USE=debug enforcing -Werror. USE=debug by definition 
>> ought to be an exception.
> 
> Now that you know that you're violating a policy, please kindly fix
> that.
I already knew about the policy. However, USE=debug is by definition meant for 
debug purposes. -Werror is helpful for debugging. There is nothing wrong with 
turning it on for debugging. The normal builds don’t have USE=debug set and 
-Werror is not used there.
> 
>> Perhaps we could have another USE flag for -Werror where it is a security 
>> feature. e.g. USE=strict-compile-checks
> 
> Perhaps people could learn that Gentoo lets them alter CFLAGS, and stop
> inventing USE flags for every flag the compiler supports.
> 
>>> 
>>> Best regards,
>>> Andrew Savchenko
>> 
>> 
> 
> -- 
> Best regards,
> Michał Górny

Re: [gentoo-dev] Changing policy about -Werror

[Richard Yao]

> On Sep 9, 2018, at 7:32 AM, Andrew Savchenko  wrote:
> 
> Hi!
> 
> Our current -Werror policy demands unconditional removal:
> https://devmanual.gentoo.org/ebuild-writing/common-mistakes/index.html#-werror-compiler-flag-not-removed
> 
> I think this is wrong, see bugs 665464, 665538 for a recent
> discussion why.
> 
> My point is that in *most* cases -Werror indeed should be removed,
> because upstream rarely can keep up with all possible configure,
> *FLAGS, compiler versions and arch combinations. But! In some cases
> — especially for security oriented software — this flag may be
> pertain and may be kept at maintainer's discretion.
> 
> The rationale is that -Werror usually points to dangerous
> situations like uninitialized variables, pointer type mismatch or
> implicit function declaration (and much more) which may lead to
> serious security implications.
> 
> So, if maintainer has enough manpower to support this flag, we
> should allow to keep it. Of course if it will cause long-standing
> troubles (e.g. bugs opened for a long time) QA should have power to
> remove it or demand its removal.
> 
> So my proposal is:
> 
> 1) Deprecate QA policy with unconditional demand of -Werror removal.
> 2) Add to devmanual's chapter on -Werror an exception clause about
> security-oriented software and maintainer's right to make final
> decision.

-Werror has caught bugs that could have resulted in data loss in ZFS in the 
past thanks to it being built in userspace as part of zdb. So it is useful for 
integrity too, not just security (although arguably, integrity is part of 
security).

Currently, sys-fs/zfs turns on -Werror when USE=debug is set. So far, nobody 
has complained about USE=debug enforcing -Werror. USE=debug by definition ought 
to be an exception.

Perhaps we could have another USE flag for -Werror where it is a security 
feature. e.g. USE=strict-compile-checks
> 
> Best regards,
> Andrew Savchenko

Re: [gentoo-dev] Gentoo i486 support

[Richard Yao]

On 08/22/2018 08:26 AM, Ben Kohler wrote:
> Hi guys,
> 
> For some time now, we've been shipping broken i486 stage3s that do not
> run on pre-i686 hardware [1].  Due to a change in catalyst [2], we no
> longer set CXXFLAGS in the default make.conf, so the x86 profiles' (imho
> wrong/broken) defaults [3] kick in.
> 
> I'd like to get this fixed, and I see 3 possible solutions, listed in
> order of my own preference:
> 
> 1) Adjust x86 profile defaults to drop the problematic -march=i686. This
> would be more in line with amd64 profiles (et al), which set no -march
> value so it can run on any hardware for this arch.
> 
> 2) Patch catalyst to start setting CXXFLAGS again.  Rather than roll
> back to exactly CXXFLAGS="${CFLAGS}" again, it's been suggested that we
> start setting COMMON_FLAGS, and CFLAGS="${COMMON_FLAGS}"
> CXXFLAGS=${COMMON_FLAGS}" etc.  I prepared such a patch a while back
> [4], which seems to work but may need a bit of updating.
> 
> 3) Drop i486 support.  We're only pretending to have support now, we
> could officially stop building these broken stages completely.
> 
> Personally I think #1 is the most technically correct and least amount
> of work.  The only result will be slightly less optimized builds for
> people who choose not to customize *FLAGS at all in make.conf.  But this
> is correct behavior.  What we have now is akin to setting -march=core2
> on amd64 stage3 and saying "oops it doesn't work on early 64bit AMD
> cpus, but oh well most people have newer and will appreciate the
> optimization".
Agreed.
> 
> Thoughts?
> 
> -Ben
> 
> [1] https://bugs.gentoo.org/654080
> [2]
> https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=b409bd9bb4b50f69a555e4e148057ade86a7ed16
> 
> [3]
> https://gitweb.gentoo.org/repo/gentoo.git/tree/profiles/arch/x86/make.defaults
> 
> [4] https://bugs.gentoo.org/575446#c4
> 




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Experimental 2-step authentication support on dev.gentoo.org

[Richard Yao]

> On Aug 9, 2018, at 4:27 AM, Michał Górny  wrote:
> 
> Hi, everyone.
> 
> Just a short notice: we've enabled experimental support for 2-step
> authentication when logging in to woodpecker via SSH.  For more details,
> see [1].
Awesome. I had no idea that the hooks for this were in place.
> 
> TL;DR: as a technical limitation, some SSH clients may start asking you
> for a password.  Using any non-empty string will work.
> 
> [1]:https://wiki.gentoo.org/wiki/Project:Infrastructure/dev.gentoo.org_2-step_authentication
> 
> -- 
> Best regards,
> Michał Górny

Re: [gentoo-dev] Idea for a new project: gentoo-libs

[Richard Yao]

> On Aug 5, 2018, at 2:35 PM, Rich Freeman  wrote:
> 
>> On Sun, Aug 5, 2018 at 2:12 PM Richard Yao  wrote:
>> 
>> 
>> Prestige is good. We have prestige from our (myself and a few others) work 
>> in upstream ZFS and Gentoo is well respected there.
> 
> Sure, but ZFS on Linux isn't a Gentoo project.
> 
> I'm not saying people who are Gentoo devs can't also do other things.
> Lots of Gentoo devs are involved in lots of stuff that gets a fair bit
> of respect. OpenRC is a big example of this.  From time to time I bump
> into signs of Gentoo on upstream projects (just today that included
> SoapySDRPlay).
> 
> Also, this gives us an opportunity to set an example, in helping to
> ensure the upstream project is maintained in a distro-friendly manner
> where Gentoo is a first class citizen.
> 
> I'm not saying that stuff like this should be banned from infra.  It
> just seems like an unnecessary constraint all-around.  It means that
> the project has to generally follow Gentoo policies/etc, and is
> limited to the tools we provide.  Just something to think about...
Which policies would those be? The upstream projects that we do have as Gentoo 
sub-projects seem to have plenty of freedom.
> 
> 
> -- 
> Rich
> 

Re: [gentoo-dev] Idea for a new project: gentoo-libs

[Richard Yao]

> On Aug 5, 2018, at 1:24 PM, Rich Freeman  wrote:
> 
>> On Sun, Aug 5, 2018 at 1:01 PM Alec Warner  wrote:
>> 
>> 
>> Part of my frustration is that seemingly "anything open source related
>> can be held in Gentoo" and I'm somewhat against that as I feel it
>> dilutes the Gentoo mission. We are here to make a distribution, not
>> maintain random libraries. If you want to do that feel free; but I
>> don't see a need for that work to be associated with Gentoo.
>> 
> 
> Honestly, other than maybe some prestige I don't really get the point
> of hosting random software in Gentoo either.  These days getting a
> repo on github or any of its 47 competitors is a few clicks.  You have
> zero overhead from a governance standpoint, and a dev can of course
> stick ebuilds in the main repository with zero interference.  It seems
> a lot cleaner from a copyright/etc standpoint as well.

Prestige is good. We have prestige from our (myself and a few others) work in 
upstream ZFS and Gentoo is well respected there. You will not hear binary 
package zealots bashing Gentoo in ZFS circles. If a sub-project that touches 
the entire OSS community can be even a tenth as effective as the efforts in ZFS 
have been in eliminating binary package zealotry, it would be well worth it.
> 
> Even openrc is hosted outside of Gentoo these days, which makes perfect sense.
> 
> With the distro as a whole it is a bit more complex, though honestly
> I'd love to see us get to a point where the whole thing can be
> SECURELY hosted entirely off-infra as well, even if we still chose to
> run our own infra.  I just see it as a way to both provide options to
> our users and ourselves.  For the latter, being able to host anything
> on an outside service means that if some component of infra goes down
> we could have mirrors already running and pulling from infra, or if
> for some reason somebody sues us or roots us or whatever we can pick
> up and move without much fuss.
> 
> Running your own wiki/bugzilla/lists/etc was about the only way to do
> things in the 90s/etc, but these days there are other options...
> 
> -- 
> Rich
> 

Re: [gentoo-dev] Idea for a new project: gentoo-libs

[Richard Yao]

> On Aug 5, 2018, at 1:01 PM, Alec Warner  wrote:
> 
> 
> 
>> On Sun, Aug 5, 2018 at 12:45 PM, Richard Yao  wrote:
>> 
>> 
>>> On Jun 23, 2018, at 6:59 AM, Alec Warner  wrote:
>>> 
>>> 
>>> 
>>>> On Sat, Jun 23, 2018 at 3:30 AM, Marty E. Plummer  
>>>> wrote:
>>>> On Sat, Jun 23, 2018 at 09:22:00AM +0200, Michał Górny wrote:
>>>> > W dniu pią, 22.06.2018 o godzinie 21∶50 -0500, użytkownik Marty E.
>>>> > Plummer napisał:
>>>> > > So, as you may be aware I've been doing some work on moving bzip2 to an
>>>> > > autotools based build. Recently I've ran into app-crypt/mhash, which is
>>>> > > in a semi-abandoned state (talking with the maintainer on twitter atm),
>>>> > > and I was thinking it may be a good idea to set up a project for 
>>>> > > keeping
>>>> > > these semi-abandoned and really-abandoned libraries and projects up to
>>>> > > date and such.
>>>> > > 
>>>> > > Basically, an upstream for packages who's upstream is either
>>>> > > uncontactable or is otherwise not accepting bug fixes and patches. So
>>>> > > far I can only think of app-crypt/mhash and app-arch/bzip2 but I'm sure
>>>> > > there are others in this state.
>>>> > > 
>>>> > 
>>>> > So in order to fix problem of semi-abandoned packages, you're creating
>>>> > an indirect herd-like entity that will soon be semi-abandoned itself
>>>> > because people will be dumping random packages into it and afterwards
>>>> > nobody will claim responsibility for them.
>>>> > 
>>>> > -- 
>>>> > Best regards,
>>>> > Michał Górny
>>>> 
>>>> No, I mean for packages which are important enough in gentoo to warrant
>>>> such treatment. For instance, every email I've tried for bzip2's
>>>> upstream bounced or recieved no reply. That, I assume, is important
>>>> enough to actually maintain and improve. Any other library which may be
>>>> as important which are as inactive would be added.
>>>> 
>>> 
>>> I suspect this might be better done in the Linux foundation itself as they 
>>> have staffing for core components that everyone is using.
>> This would put decision making power into the hands of bureaucrats. I would 
>> rather it remain in a community of volunteers.
> 
> Meh, it doesn't hurt to ask there about interest (they certainly fund 
> development of other components.) Its not like they have to accept, or that 
> declining somehow inhibits this development.
> 
> Part of my frustration is that seemingly "anything open source related can be 
> held in Gentoo" and I'm somewhat against that as I feel it dilutes the Gentoo 
> mission. We are here to make a distribution, not maintain random libraries. 
> If you want to do that feel free; but I don't see a need for that work to be 
> associated with Gentoo.

This could just be done as a downstream effort that carries patches without a 
subproject, but structuring it as a subproject would be a call for 
collaboration with other distributions, which would ultimately benefit us.

>  
>> 
>> I consider upstream development efforts by Gentoo developers to be 
>> beneficial to Gentoo. Nothing makes fixing an issue in Gentoo at upstream a 
>> priority quite like it affecting a key upstream developer in his day to day 
>> life.
>> 
>> 
>> Also, the Linux Foundation is not embarking on such a project and we clearly 
>> have someone willing to try, so I say that we should go for it. Having 
>> people that wish to take a more active role in upstream development would 
>> not make us any worse off. It is their time to volunteer, so it is not like 
>> they will volunteer it for something else if we discourage them.

Re: [gentoo-dev] Idea for a new project: gentoo-libs

[Richard Yao]

> On Jun 23, 2018, at 9:05 AM, Jonas Stein  wrote:
> 
>> On 2018-06-23 04:57, Marty E. Plummer wrote:
>>> On Fri, Jun 22, 2018 at 09:50:50PM -0500, Marty E. Plummer wrote:
>>> So, as you may be aware I've been doing some work on moving bzip2 to an
>>> autotools based build. Recently I've ran into app-crypt/mhash, which is
>>> in a semi-abandoned state (talking with the maintainer on twitter atm),
>>> and I was thinking it may be a good idea to set up a project for keeping
>>> these semi-abandoned and really-abandoned libraries and projects up to
>>> date and such.
>>> 
>>> Basically, an upstream for packages who's upstream is either
>>> uncontactable or is otherwise not accepting bug fixes and patches. So
>>> far I can only think of app-crypt/mhash and app-arch/bzip2 but I'm sure
>>> there are others in this state.
>>> 
>> Or... call it proxy-upstream, to be in line with the current proxy-maint
>> setup?
> 
> Please do not call it proxy-*.
> The invented word proxymaintainer and proxiedmaintainer are not usefull.
> They get always mixed, and are not understood outside of Gentoo.
> assistant developer or trainee developer would have been much more useful.

Until we have a better name, why not call it the GPL project? GPL meaning:

Gentoo
POSIX
Libraries

Misunderstandings from the obvious acronym collision aside, I think the name 
GPL Project would be more appealing to outsiders than “proxy-libs”, which I 
agree would be misunderstood in the worst possible ways.
> 
> Beside the naming I like the idea, that you want to take care for all
> abandoned libs.
> 
> Please note, that you can not generate more manpower by creating a
> project. In 2015 I calculated
In the case of creating a new upstream for key libraries shared by 
distributions, I disagree. As long as other distribution maintainers get on 
board, the deduplication of effort will result in more man power.
> 
> =
> 
> (Number of developers) / (Number of Projects) < 1.4
> 
> =
> 
> Which explains, why most projects today are run by mostly one active person.
> 
> If you find an important library, where upstream is dead, fork it and
> take responsibility for it.
I will call this point 1 of yours.

That sounds like what this project is intended to do.
> 
> It makes no sense to make a pool of dead and important software and
> delegate the responsibility to a team where nobody really knows the
> software.
I will call this point 2 of yours.

It depends on the importance of the software.
> 
> Better pick a library, communicate with maintainers of the other
> distributions and fork it. Keep the library alive in the fork.
I feel like this is the same as 1.
> 
> It is important for the security to let dead projects die.
I feel like this is 2, and that it contradicts 1.
> 
> -- 
> Best,
> Jonas
> 

Re: [gentoo-dev] Idea for a new project: gentoo-libs

[Richard Yao]

> On Jun 23, 2018, at 6:59 AM, Alec Warner  wrote:
> 
> 
> 
>> On Sat, Jun 23, 2018 at 3:30 AM, Marty E. Plummer  
>> wrote:
>> On Sat, Jun 23, 2018 at 09:22:00AM +0200, Michał Górny wrote:
>> > W dniu pią, 22.06.2018 o godzinie 21∶50 -0500, użytkownik Marty E.
>> > Plummer napisał:
>> > > So, as you may be aware I've been doing some work on moving bzip2 to an
>> > > autotools based build. Recently I've ran into app-crypt/mhash, which is
>> > > in a semi-abandoned state (talking with the maintainer on twitter atm),
>> > > and I was thinking it may be a good idea to set up a project for keeping
>> > > these semi-abandoned and really-abandoned libraries and projects up to
>> > > date and such.
>> > > 
>> > > Basically, an upstream for packages who's upstream is either
>> > > uncontactable or is otherwise not accepting bug fixes and patches. So
>> > > far I can only think of app-crypt/mhash and app-arch/bzip2 but I'm sure
>> > > there are others in this state.
>> > > 
>> > 
>> > So in order to fix problem of semi-abandoned packages, you're creating
>> > an indirect herd-like entity that will soon be semi-abandoned itself
>> > because people will be dumping random packages into it and afterwards
>> > nobody will claim responsibility for them.
>> > 
>> > -- 
>> > Best regards,
>> > Michał Górny
>> 
>> No, I mean for packages which are important enough in gentoo to warrant
>> such treatment. For instance, every email I've tried for bzip2's
>> upstream bounced or recieved no reply. That, I assume, is important
>> enough to actually maintain and improve. Any other library which may be
>> as important which are as inactive would be added.
>> 
> 
> I suspect this might be better done in the Linux foundation itself as they 
> have staffing for core components that everyone is using.
This would put decision making power into the hands of bureaucrats. I would 
rather it remain in a community of volunteers.

I consider upstream development efforts by Gentoo developers to be beneficial 
to Gentoo. Nothing makes fixing an issue in Gentoo at upstream a priority quite 
like it affecting a key upstream developer in his day to day life.

Also, the Linux Foundation is not embarking on such a project and we clearly 
have someone willing to try, so I say that we should go for it. Having people 
that wish to take a more active role in upstream development would not make us 
any worse off. It is their time to volunteer, so it is not like they will 
volunteer it for something else if we discourage them.
> 
> -A
> 
> 
> 

[gentoo-dev] Cross compilation tracker bug

[Richard Yao]

I have opened a tracker bug for cross compilation issues:

https://bugs.gentoo.org/662714

Bugs on packages that fail to build with crossdev and patches for those
bugs are more than welcome. :)



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: moving default location of portage tree

[Richard Yao]

> On Jul 11, 2018, at 6:24 PM, Rich Freeman  wrote:
> 
>> On Wed, Jul 11, 2018 at 6:11 PM Richard Yao  wrote:
>> 
>> Is it a violation of the FHS? /usr is for readonly data and the portage tree 
>> is generally readonly, except when being updated. The same is true of 
>> everything else in /usr.
>> 
> 
> It is application metadata.  It belongs in /var.  No other packages
> write to /usr when they're doing internal updates.  Obviously you need
> a writable /usr to actually install package changes, but that
> shouldn't be necessary just to sync the repository.
> 
> I was asking around and it seems like most distros stick their
> repositories in /var/lib.  I can't imagine that too many would have
> even considered sticking them in /usr.
I would consider the package manager to be special in that it is a step of the 
system update process, but I agree that it could be nicer to have in /var.

We have a problem using /var/lib because /var/lib/portage is already in use. I 
guess /var/portage is not  a terrible choice.
> 
>> I am confused as to how we only now realized it was a FHS violation when it 
>> has been there for ~15 years. I was under the impression that /usr was the 
>> correct place for it.
> 
> It has certainly been pointed out in the past.  Nothing was changed
> for the same reason that nothing will probably be changed this time -
> people don't like change and the people who know better just slowly
> patch around Gentoo's oddities.  Somebody was just posting a manifesto
> about deploying more experimental technologies, and here we can't move
> a repository out of /usr.
> 
> And if nothing else, can we at least move /usr/portage/distfiles
> someplace else?  Surely you have to agree that this doesn't belong in
> usr, or nested in the middle of a repository?
Well, if it is changed during system updates, then the same logic applies, but 
quite honestly, I never liked having the distfiles or packages directories 
there. You can have unnecessary headaches when mounting inside a mount point 
other than /.
> 
> -- 
> Rich
> 

Re: [gentoo-dev] rfc: moving default location of portage tree

[Richard Yao]

> On Jul 11, 2018, at 6:23 PM, Michał Górny  wrote:
> 
> W dniu śro, 11.07.2018 o godzinie 18∶11 -0400, użytkownik Richard Yao
> napisał:
>>>> On Jul 11, 2018, at 4:43 PM, Rich Freeman  wrote:
>>>> 
>>>> On Wed, Jul 11, 2018 at 4:34 PM Richard Yao  wrote:
>>>> 
>>>> On my system, /usr/portage is a separate mountpoint. There is no need to 
>>>> have on,h top level directories be separate mountpoints.
>>> 
>>> It makes sense to follow FHS.  Sure, I can work around poor designs by
>>> sticking mount points all over the place, or manually setting my
>>> config to put stuff in sane locations.  It makes more sense to put all
>>> the volatile stuff in /var, than to mix it up all over the place and
>>> get users to set up separate mountpoints to make up for it.
>> 
>> Is it a violation of the FHS? /usr is for readonly data and the portage tree 
>> is generally readonly, except when being updated. The same is true of 
>> everything else in /usr.
>> 
>> I am confused as to how we only now realized it was a FHS violation when it 
>> has been there for ~15 years. I was under the impression that /usr was the 
>> correct place for it.
>>> 
> 
> And we're back to the usual Gentoo argument of 'it was like this for
> N years'.  So FYI, something 'being there for ~15 years' doesn't make it
> right.  It only means that:
> 
> a. Gentoo devs were wrong 15 years ago.
> 
> b. Gentoo devs are still wrong today.
> 
> c. Gentoo devs can't manage to make such a simple change because they're
> too concerned about hurting somebody's feelings about a path.
This does not answer my question. Is it really a FHS violation? The contents of 
/usr changes when doing updates using the system package manager. When not 
doing updates, it really is readonly and the FHS says that /usr is for readonly 
things. I do not see how it is different from anything else in /usr.

I have been thinking that having it there was compliant for years and honestly, 
I don’t see how it is not complaint. Saying it is not compliant is not an 
explanation.
> 
> -- 
> Best regards,
> Michał Górny

Re: [gentoo-dev] rfc: moving default location of portage tree

[Richard Yao]

> On Jul 11, 2018, at 4:43 PM, Rich Freeman  wrote:
> 
>> On Wed, Jul 11, 2018 at 4:34 PM Richard Yao  wrote:
>> 
>> On my system, /usr/portage is a separate mountpoint. There is no need to 
>> have on,h top level directories be separate mountpoints.
> 
> It makes sense to follow FHS.  Sure, I can work around poor designs by
> sticking mount points all over the place, or manually setting my
> config to put stuff in sane locations.  It makes more sense to put all
> the volatile stuff in /var, than to mix it up all over the place and
> get users to set up separate mountpoints to make up for it.

Is it a violation of the FHS? /usr is for readonly data and the portage tree is 
generally readonly, except when being updated. The same is true of everything 
else in /usr.

I am confused as to how we only now realized it was a FHS violation when it has 
been there for ~15 years. I was under the impression that /usr was the correct 
place for it.
> 
> If somebody is doing a new Gentoo install, why would they want to put
> the repository in /usr, and nest a few GB of distfiles inside of the
> repo?  Why should that be the place we direct them?  There is no
> history for them.  A brand new install should put things in the most
> logical place.
> 
> By all means let existing users decide whether to move stuff.  I'm
> sure we have plenty of users with make.conf in /etc/.
> 
> -- 
> Rich
> 

Re: [gentoo-dev] rfc: moving default location of portage tree

[Richard Yao]

> On Jul 11, 2018, at 4:42 PM, William Hubbs  wrote:
> 
>> On Wed, Jul 11, 2018 at 04:25:20PM -0400, Richard Yao wrote:
>>> On 07/11/2018 03:29 AM, Jory A. Pratt wrote:
>>>> On 07/10/18 16:35, M. J. Everitt wrote:
>>>>> On 10/07/18 21:09, William Hubbs wrote:
>>>>>> On Mon, Jul 09, 2018 at 03:54:35PM -0700, Zac Medico wrote:
>>>>>>> On 07/09/2018 03:27 PM, M. J. Everitt wrote:
>>>>>>>> On 09/07/18 23:12, Zac Medico wrote:
>>>>>>>>> On 07/09/2018 02:34 PM, Kristian Fiskerstrand wrote:
>>>>>>>>> I'd mostly argue any such change should only affect new systems
>>>>>>>>> 
>>>>>>>> Yes, changing defaults for existing systems would be annoying.
>>>>>>>> 
>>>>>>>> My recommendation is to have catalyst set the new defaults in the stage
>>>>>>>> tarballs.
>>>>>>>> 
>>>>>>>> When sys-apps/portage changes its internal defaults, I'd like for the
>>>>>>>> upgrade process to call a tool that generates configuration files when
>>>>>>>> necessary to ensure that the existing paths remain constant.
>>>>>>> I think it should be possible for RelEng to make a start on catalyst
>>>>>>> updates - is there anything that would inhibit going ahead with this,
>>>>>>> potentially?
>>>>>> No, nothing. Whatever catalyst puts it the default config will become
>>>>>> our new default.
>>>>> I would still like to see notice about what the new defaults are and how
>>>>> to migrate current systems to them.
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> William
>>>>> 
>>>>>> -- 
>>>>>> Thanks,
>>>>>> Zac
>>>>>> 
>>>>> 
>>>>> 
>>>> I'd like to propose that further to the discussion here on the -dev
>>>> mailing list, the Council discuss and make a firm proposal on the new
>>>> default paths, and then RelEng can make the appropriate updates to the
>>>> catalyst builds. A news item can be compiled, with an appropriate wiki
>>>> article perhaps on migration strategy (I may volunteer to format such a
>>>> page with some appropriate guidance).
>>>> Regards,
>>>> Michael / veremitz.
>>>> 
>>> This is a mess, many systems are setup with portage already on a
>>> seperate partition for reasons. What advantage does it provide to move
>>> the tree now after all these years? I have seen nothing more then lets
>>> do this cause I like the ideal lately and it is getting old, there is no
>>> benefit that would justify moving the tree or many other changes that
>>> are being made in Gentoo lately.
>> 
>> People who want to move it could just set PORTDIR in make.conf. I don't
>> see any reason to move it either.
> 
> Actually, I believe that PORTDIR is becoming a thing of the past.
I used to use it 5 years ago. If it does not work due to regressions, we should 
fix that.
> 
> Also, the default definitely should not be on /usr per fhs. This would
> allow /usr to be mounted read only.
> This doesn't affect things like the example above where /usr/portage is
> a mount point.
> 
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> 

Re: [gentoo-dev] rfc: moving default location of portage tree

[Richard Yao]

> On Jul 11, 2018, at 11:56 AM, Rich Freeman  wrote:
> 
>> On Wed, Jul 11, 2018 at 11:36 AM Raymond Jennings  wrote:
>> 
>> I do think it would be a wise idea to "grandfather" the current layout
>> for awhile.
>> 
> 
> I don't see why we would ever stop supporting it, at least in general.
> Maybe if some day somebody had a solution for a read-only /usr with
> signature checking that might require portage to be mounted elsewhere,
> but I don't ever see that becoming the default.
> 
> Portage just looks for the repository where you tell it to.  If you
> tell it that the repository is in /var, it will use it.  If you put it
> in /tmp, that's fine too.
> 
> This is just about the default, which should follow FHS.  The case of
> separate mounts is exactly why /usr is a bad spot - the access
> patterns for something like the repository have far more in common
> with /var than /usr.

On my system, /usr/portage is a separate mountpoint. There is no need to have 
on,h top level directories be separate mountpoints.
> 
> -- 
> Rich
> 

Re: [gentoo-dev] rfc: moving default location of portage tree

[Richard Yao]

On 07/11/2018 03:29 AM, Jory A. Pratt wrote:
> On 07/10/18 16:35, M. J. Everitt wrote:
>> On 10/07/18 21:09, William Hubbs wrote:
>>> On Mon, Jul 09, 2018 at 03:54:35PM -0700, Zac Medico wrote:
 On 07/09/2018 03:27 PM, M. J. Everitt wrote:
> On 09/07/18 23:12, Zac Medico wrote:
>> On 07/09/2018 02:34 PM, Kristian Fiskerstrand wrote:
>>> I'd mostly argue any such change should only affect new systems
>>>
>> Yes, changing defaults for existing systems would be annoying.
>>
>> My recommendation is to have catalyst set the new defaults in the stage
>> tarballs.
>>
>> When sys-apps/portage changes its internal defaults, I'd like for the
>> upgrade process to call a tool that generates configuration files when
>> necessary to ensure that the existing paths remain constant.
> I think it should be possible for RelEng to make a start on catalyst
> updates - is there anything that would inhibit going ahead with this,
> potentially?
 No, nothing. Whatever catalyst puts it the default config will become
 our new default.
>>> I would still like to see notice about what the new defaults are and how
>>> to migrate current systems to them.
>>>
>>>
>>> Thanks,
>>>
>>> William
>>>
 -- 
 Thanks,
 Zac

>>>
>>>
>> I'd like to propose that further to the discussion here on the -dev
>> mailing list, the Council discuss and make a firm proposal on the new
>> default paths, and then RelEng can make the appropriate updates to the
>> catalyst builds. A news item can be compiled, with an appropriate wiki
>> article perhaps on migration strategy (I may volunteer to format such a
>> page with some appropriate guidance).
>> Regards,
>> Michael / veremitz.
>>
> This is a mess, many systems are setup with portage already on a
> seperate partition for reasons. What advantage does it provide to move
> the tree now after all these years? I have seen nothing more then lets
> do this cause I like the ideal lately and it is getting old, there is no
> benefit that would justify moving the tree or many other changes that
> are being made in Gentoo lately.

People who want to move it could just set PORTDIR in make.conf. I don't
see any reason to move it either.
> 
> 
> 




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH v3 00/12] GLEP 63 update

[Richard Yao]

> On Jul 5, 2018, at 4:53 PM, Michał Górny  wrote:
> 
> Hi,
> 
> Here's third version of the patches.  I've incorporated the feedback
> so far and reordered the patches (again) to restore their
> degree-of-compatibility order.  The full text is included below.
> 
> 
> Michał Górny (12):
>  glep-0063: Use 'OpenPGP' as appropriate
>  glep-0063: RSAv4 -> OpenPGP v4 key format
>  glep-0063: 'Gentoo subkey' → 'Signing subkey'
>  glep-0063: Root key → primary key
>  glep-0063: Split out the signing subkey into a separation point
>  glep-0063: Explain minimal & recommended sections
>  glep-0063: Change the recommended RSA key size to 2048 bits
>  glep-0063: Allow ECC curve 25519 keys
>  glep-0063: Stop recommending DSA subkeys
>  glep-0063: Make 2-yearly expiration term mandatory
>  glep-0063: Require renewal 2 weeks before expiration
>  glep-0063: Disallow using DSA keys
> 
> glep-0063.rst | 97 +--
> 1 file changed, 64 insertions(+), 33 deletions(-)
> 
> 
> ---
> GLEP: 63
> Title: Gentoo OpenPGP policies
> Author: Robin H. Johnson ,
>Andreas K. Hüttel ,
>Marissa Fischer 
> Type: Standards Track
> Status: Final
> Version: 2
> Created: 2013-02-18
> Last-Modified: 2018-07-05
> Post-History: 2013-11-10
> Content-Type: text/x-rst
> ---
> 
> Credits
> ===
> 
> Many developers and external sources helped in this GLEP.
> 
> Abstract
> 
> 
> This GLEP provides both a minimum requirement and a recommended set of
> OpenPGP key management policies for the Gentoo Linux distribution.
> 
> Changes
> ===
> 
> v2
>  The distinct minimal and recommended expirations have been replaced
>  by a single requirement. The rules have been simplified to use
>  the same time of 2 years for both the primary key and subkeys.
> 
>  An additional rule requesting key renewal 2 weeks before expiration
>  has been added. This is in order to give services and other developers time
>  to refresh the key.
> 
>  The usage of DSA keys has been disallowed.
> 
> v1.1
>  The recommended RSA key size has been changed from 4096 bits
>  to 2048 bits to match the GnuPG recommendations [#GNUPG-FAQ-11-4]_.
>  The larger recommendation was unjustified and resulted in people
>  unnecessarily replacing their RSA-2048 keys.
> 
>  Minimal specification has been amended to allow for ECC keys.
> 
>  The option of using DSA subkey has been removed from recommendations.
>  The section now specifies a single recommendation of using RSA.
> 
> Motivation
> ==
> 
> Given the increasing use and importance of cryptographic protocols in internet
> transactions of any kind, unified requirements for OpenPGP keys used in Gentoo
> Linux development are sorely needed.  This document provides both a set of
> bare minimum requirements and a set of best practice recommendations for
> the use of GnuPG (or other OpenPGP providers) by Gentoo Linux developers.
> It is intended to provide a basis for future improvements such as, e.g.,
> consistent ebuild or package signing and verifying by end users.
> 
> Specifications for OpenPGP keys
> ===
> 
> Bare minimum requirements
> -
> This section specifies obligatory requirements for all OpenPGP keys used
> to commit to Gentoo. Keys that do not conform to those requirements can
> not be used to commit.
> 
> 1. SHA2-series output digest (SHA1 digests internally permitted),
>   256bit or more::
> 
>   personal-digest-preferences SHA256
> 
> 2. Signing subkey that is different from the primary key, and does not
>   have any other capabilities enabled.
> 
> 3. Primary key and the signing subkey are both of type EITHER:
> 
>   a. RSA, >=2048 bits (OpenPGP v4 key format or later only)
> 
>   b. ECC curve 25519
> 
> 4. Expiration date on key and all subkeys set to at most 2 years
> 
> 5. Key expiration date renewed at least 2 weeks before the previous
>   expiration date.
> 
> 6. Upload your key to the SKS keyserver rotation before usage!
> 
> Recommendations
> ---
> This section specifies the best practices for Gentoo developers.
> The developers should follow those practices unless there is a strong
> technical reason not to (e.g. hardware limitations, necessity of replacing
> their primary key).
> 
> 1. Copy ``/usr/share/gnupg/gpg-conf.skel`` to ``~/.gnupg/gpg.conf``, append
>   the following block::
That file no longer exists.
> 
>   keyserver pool.sks-keyservers.net
This is less secure than the default according to K_F in #gentoo-infra.
> 
>   emit-version
K_F indicated that this harms security too.
> 
>   default-recipient-self
> 
>   # -- All of the below portion from the RiseUp.net OpenPGP best 
> practices, and
>   # -- many of them are also in the Debian GPG documentation.
> 
>   # when outputting certificates, view user IDs distinctly from keys:
>   fixed-list-mode
> 
>   # long keyids are more collision-resistant than short keyids (it's 
> trivial to 

Re: [gentoo-dev] RFC: making ebuilds that provide mkfs.* programs include kernel config checks for fcaps (or other xattrs)

[Richard Yao]

On 06/29/2018 11:43 AM, Zac Medico wrote:
> Hi,
> 
> As suggested by Wojciech Myrda in bug 659486 [1], it would be helpful if
> sys-fs/e2fsprogs would use the linux-info eclass to warn if the kernel
> configuration doesn't include CONFIG_EXT4_FS_SECURITY=y when the user
> has expressed a desire to use file capabilities. This idea can be
> extended to any ebuild that provides an mkfs.* program. Thoughts?
> 
> [1] https://bugs.gentoo.org/659486#c11
> 

I like this idea. We should do it.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Hostile takeover of our github mirror. Don't use ebuild from there until new warning!

[Richard Yao]

> On Jun 28, 2018, at 8:46 PM, Richard Yao  wrote:
> 
> 
>> On Jun 28, 2018, at 5:15 PM, Francisco Blas Izquierdo Riera (klondike) 
>>  wrote:
>> 
>> Hi!
>> 
>> I just want to notify that an attacker has taken control of the Gentoo
>> organization in Github and has among other things replaced the portage
>> and musl-dev trees with malicious versions of the ebuilds intended to
>> try removing all of your files.
>> 
>> Whilst the malicious code shouldn't work as is and GitHub has now
>> removed the organization, please don't use any ebuild from the GitHub
>> mirror ontained before 28/06/2018, 18:00 GMT  until new warning.
> Is the attacker using the account “gentoogang”?

Nevermind. After reading other mailing list threads, it is clear to me that he 
was the attacker. :/
>> 
>> Sincerely,
>> Francisco Blas Izquierdo Riera (klondike)
>> Gentoo developer.
>> 
>> 
> 
> 

Re: [gentoo-dev] Hostile takeover of our github mirror. Don't use ebuild from there until new warning!

[Richard Yao]

> On Jun 28, 2018, at 5:15 PM, Francisco Blas Izquierdo Riera (klondike) 
>  wrote:
> 
> Hi!
> 
> I just want to notify that an attacker has taken control of the Gentoo
> organization in Github and has among other things replaced the portage
> and musl-dev trees with malicious versions of the ebuilds intended to
> try removing all of your files.
> 
> Whilst the malicious code shouldn't work as is and GitHub has now
> removed the organization, please don't use any ebuild from the GitHub
> mirror ontained before 28/06/2018, 18:00 GMT  until new warning.
Is the attacker using the account “gentoogang”?
> 
> Sincerely,
> Francisco Blas Izquierdo Riera (klondike)
> Gentoo developer.
> 
> 

Re: [gentoo-dev] glibc-2.26 just went stable (on amd64)

[Richard Yao]

> On Jun 2, 2018, at 3:47 PM, Andreas K. Huettel  wrote:
> 
> As a heads-up, glibc-2.26 just went stable on amd64. 
> 
> If you still have open bugs, they now mutate from "doesn't build with 
> glibc-2.26" to "doesn't build, treecleaning candidate".
Not necessarily. If it builds on Gentoo FreeBSD, we could just mask it in 
Gentoo Linux. I imagine such cases are rare though.
> 
> Cheers, 
> Andreas
> 
> -- 
> Andreas K. Hüttel
> dilfri...@gentoo.org
> Gentoo Linux developer 
> (council, toolchain, perl, libreoffice, comrel)

Re: [gentoo-dev] Monthly x11@ project status for June 2018

[Richard Yao]

> On Jun 1, 2018, at 2:58 PM, Matt Turner  wrote:
> 
> x11@ is currently assigned or cc'd on 176 bugs. This number is down from
> 222 on April 1st and more than 412 in February 2015 (I reported this on
> #gentoo-desktop after closing out a bunch of bugs that day).
Awesome!
> 
> 
> == Fix x11-base/xorg-server suid/systemd situation ==
> https://bugs.gentoo.org/635102
> 
> Looks like these bugs are sorted out. There are a few issues in
> conjunction with Mesa 18.x, so I'm going to hold off on unmasking
> xorg-server-1.20 until those are sorted. I think fixes are at least in
> progress for all of those, so I expect we're in good shape and will have
> unmasked 1.20 by next month.
> 
> == Convert media-libs/mesa ebuild to build with Meson ==
> https://bugs.gentoo.org/652762
> 
> Lots of progress. floppym made the necessary changes to meson.eclass to
> let Mesa's build system use a different *-llvm-config per-ABI by always
> using a cross file. That lets multilib builds work.
> 
> There are a couple of (hopefully minor) configuration options missing
> from Mesa's meson.build, so I'm targeting next month for the transition
> of mesa-.ebuild to Meson. I plan to ship Mesa 18.2 build with Meson.
> 
> == Update packages to depend on x11-base/xorg-proto ==
> https://bugs.gentoo.org/656250
> 
> Arfrever continued transitioning whole categories at a time! Big thanks
> to him!
> 
>   Apr 1May 7  Jun 1
> Packages depending on x11-proto/*:  530 ->   432 ->  35 [1]
> ebuilds  depending on x11-proto/*: 1908 ->  1503 ->  59 [2]
> 
> Just some games-*/* and kde-frameworks/* packages remain to be updated.
> 
> == media-libs/vulkan-* split packages ==
> Nick (sarnex) had lots of fun this month splitting packages out of
> media-libs/vulkan-loader:
> 
>media-libs/vulkan-layers
>media-libs/vulkan-toolsmedia-libs/vulkan-headers
> 
> Upstream continues to churn things and doesn't seem to have a good grasp
> of how Linux distributions operate.  Nick's done a good job keeping on
> top of upstream changes.
> 
> == Tree cleaning ==
> Removed:
>x11-proto/printproto
>x11-libs/libXp
>x11-libs/xpyb
> Given Last Rites:
>x11-libs/libXfont
>app-i18n/fbiterm (reverse dependency of libXfont)
> 
> == Add and require glvnd ==
> https://bugs.gentoo.org/606924
> https://github.com/NVIDIA/libglvnd
> 
> Package added under package.mask:
> 
># New package. Needs to interact with media-libs/mesa and
># x11-drivers/nvidia-drivers. Work in progress.
>media-libs/libglvnd
> 
> nvidia-drivers-375 and newer support glvnd, so I'll have to figure out
> how to make things work. I guess we'll have to keep eselect-opengl for
> older nvidia-drivers, which means updating it to support glvnd (instead
> of removing it)
> 
> == Drop app-eselect/eselect-mesa ==
> https://bugs.gentoo.org/576334
> 
> Removed from mesa-.ebuild and mesa-18.1.0-r1.ebuild. Once older
> versions are gone from the tree we can give eselect-mesa last rites.
> 
> Without eselect-mesa, the Gallium i915 and swrast drivers are installed
> if USE=gallium; otherwise the classic versions are installed. I suspect
> this is good enough.
> 
> == Fix/Remove OpenCL ==
> https://bugs.gentoo.org/546320
> https://bugs.gentoo.org/647330
> 
> No progress.
> 
> == Clean out x11 overlay ==
> https://gitweb.gentoo.org/proj/x11.git/
> 
> No progress.
> 
> [1] git grep x11-proto | cut -d '/' -f -2 | grep -vE 
> 'xorg-proto|metadata|profiles' | sort -u  | wc -l
> [2] git grep x11-proto | grep -vE 'xorg-proto|metadata|profiles' | wc -l

Re: [gentoo-dev] New Portage fork: sys-apps/portage-mgorny

[Richard Yao]

> On May 22, 2018, at 4:35 PM, Michał Górny  wrote:
> 
> W dniu sob, 19.05.2018 o godzinie 18∶53 +0300, użytkownik Consus
> napisał:
>> Okay, this
>> 
>>https://github.com/mgorny/portage-mgorny/issues/15
>> 
>> is a goddamn piece of sanity that Portage requires for a long time and
>> and it worth some $20 donations. Where to send moneyz?
>> 
> 
> Thanks.  However, this really isn't something that deserves money.  I'm
> pretty sure you'll find a better use for it.

I suggest that you make an amazon wishlist and let people buy you things from 
there. Put some books on it that would be useful for development.
> 
> -- 
> Best regards,
> Michał Górny
> 
> 

[gentoo-dev] Most of my packages up for grabs

[Richard Yao]

My past jobs have worn me out, which combined with upstream work, caused
me to become minimally active for some time. I am in the middle of some
changes to my life that will likely change that, but I don't use most of
the packages that I maintain anymore and they have gotten very little
attention since they fell into disuse by me.

The packages that I do use have enough outstanding bugs both downstream
and upstream to keep me occupied from now until doomsday, so it would be
better that the packages that I do not use were maintained by people who
use them.

Consequently, the following packages are officially up for grabs:

app-editors/bvi
dev-python/eliot
dev-python/treq
net-dns/https_dns_proxy
net-libs/libiscsi
sys-devel/heirloom-devtools
sys-freebsd/ubin-wrappers
sys-freebsd/virtio-kmod
sys-libs/nss-usrfiles
www-client/dooble

The BSD team likely would want the two sys-freebsd ones.

I have omitted sys-kernel/genkernel, app-arch/lz4, sys-kernel/spl,
sys-fs/zfs-kmod and sys-fs/zfs with whom I have co-maintainers. Those
are not up for grabs (nor could I put genkernel up for grabs).
Furthermore, I am active upstream in 3/5 of those, so I would keep them
even if I did not have co-maintainers. However, if mgorny would like to
be the sole lz4 maintainer, I would be happy to step aside. The change
would be long overdue.

I have also omitted app-emulation/crossover-bin, but I would appreciate
a co-maintainer for it, which gets updated as I use it. That happens
about once per year on average, although less lately because I have far
less need for Windows software these days. I can arrange with
codeweavers for the person willing to co-maintain it to get free access
to it. If interest, please ping me in freenode IRC regarding this. I am
far less responsive via email than I am via IRC these days.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] usr merge

[Richard Yao]

>> On Apr 6, 2016, at 4:43 PM, William Hubbs <willi...@gentoo.org> wrote:
>> 
>>> On Wed, Apr 06, 2016 at 11:52:52AM -0400, Richard Yao wrote:
>>> On 04/06/2016 10:58 AM, M. J. Everitt wrote:
>>> What, if any, is the benefit of squashing /usr out of the equation? I
>>> happen to have a few workstations that load their /usr off an NFS share
>>> presently, with some bodgery-workarounds I did pre the udev notification
>>> about initramfs's which I have never got around to implementing
>>> (although I'm pretty sure I have the tools now to do, along with UUIDs
>>> for boot media).
>> 
>> The idea in Solaris is to enable atomic updates via the /usr mount
>> without touching data files in /etc or temporary files in /var. Usually,
>> this would be done on reboot and could be propagated to many systems
>> either via /usr on NFS or ZFS send/recv.
>> 
>> This works well on Solaris because both software versions are pegged
>> (such that file formats in /etc are stable) in favor of backported fixes
>> and the FHS does not change across major OS versions. The same goes for
>> RHEL.
> 
> Also, there are other benefits to the /usr merge [1].

Are they worth breaking existing systems that are configured the one way we all 
know things will break if this is forced? If not, a USE flag would work.

As for those benefits, they do little for {/usr,}/sbin vs {/usr,}/bin, which is 
where the incompatibilities tend to live. I encountered one of these in 
powertop the other day (patch pending). The benefits of being able to access 
things from both places are somewhat exaggerated given that compatibility among 
systems has long required searching $PATH and likely always will.

> Note, we are not
> talking about squashing /usr out of the equasion, but merging /bin,
> /sbin and /lib* into their counterparts in /usr and creating symlinks in
> the root directory pointing to the counterparts in /usr.

While one guy did the reverse (and the reverse ought to be okay for those that 
want to do that), no one appears to think that adopting the reverse is what is 
being suggested. Having this sort of clarity on whether forcing this on 
everyone via baselayout update, just providing the option for those who want it 
or some combination of the two (e.g. a long transition period in which both are 
supported) is being discussed would be nice though. This is not a Boolean 
decision.

>> Gentoo systems managed this way will suffer from multiple problems:
>> 
>> * Software updates that change the configuration file format without
>> supporting the older format will break.
>> 
>> * Software updates that change the boot scripts will break.
>> 
>> * Future baselayout updates will not be able to touch anything outside
>> of /usr and anything requiring such things be touched will break.
>> 
>> * An update to /usr that adds new software will fail to include things
>> outside of /usr, like the boot scripts and configuration files.
>> 
>> * The package database will fall out of sync with /usr (or be broken
>> period). Presumably, if you are updating this way, you should expect the
>> package database to be broken.
>> 
>> These are likely to be mostly fixable, but I do not think we have a plan
>> in place to fix them right now. The general staleness of Solaris and
>> RHEL handle the first 3 issues for them for free.
>> 
>> I have not looked at the specifics of how Solaris handles the 4th, but I
>> know that SMF in OpenSolaris descendents will update manifests on first
>> boot into a new boot environment. That suggests to me that the Solaris
>> boot scripts handle it by comparing /etc with /usr.
>> 
>> As for the 5th, the package database is not broken in Solaris zones
>> where the /usr merge is leveraged to enable easy updates. However, I do
>> not know how updating all zones works when zones have independently
>> installed software. It might be that the software is installed in
>> /usr/local inside the zone and conflicts are the user's problem, but it
>> has been so long since I used an illumos distribution (which is
>> descended from OpenSolaris) that I do not remember.
> 
> I don't think any of these issues are issues that Gentoo systems
> managed like this do not already have.

That is my point.

> If you are mounting /usr from nfs right now, for example,
> things are worse, because you also have to worry about whether packages
> split their installations between /usr/lib*->/lib* and
> /usr/{,s}bin->/{s,}bin.

Only a masochist would want to do this right now. There are saner ways of doing 
things with the legacy layout than the Solaris way that depends

Re: [gentoo-dev] GitHub GPG Signature Verification

[Richard Yao]

On 04/06/2016 01:52 PM, NP-Hardass wrote:
> Greetings all,
> 
> As of yesterday, GitHub now supports GPG signature verification [1].
> As a result, when viewed through the GitHub mirror, all commits now
> have a widget that displays whether the GPG signature has been
> verified (via GitHub).  To help facilitate this, it is highly
> recommend that all developers who use GitHub update their accounts to
> include their GPG public key [2] so that their commits can register as
> verified, where applicable.
> 
> Just a disclaimer, I'm not endorsing the use of GitHub, nor advocating
> for it, simply indicating a new feature which it would be nice to see
> Gentoo developers on GitHub taking advantage of.
> 
> 

Thanks for letting us know. :)



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] usr merge

[Richard Yao]

On 04/06/2016 12:33 PM, Richard Yao wrote:
> On 04/06/2016 12:20 PM, Alexis Ballier wrote:
>> On Wednesday, April 6, 2016 6:06:35 PM CEST, Richard Yao wrote:
>>
>>> That is unless you put per-system state in /usr/local, do symlinks to it
>>> in / and mount /usr/local as part of system boot, which is the other way
>>> of doing this. I have seen a variant of this done in asuswrt-merlin on
>>> routers.
>>
>> This doesnt seem to have anything to do with what I was describing.
>>
>> Another option I'm using a lot is nfsroot. This doesn't have the same
>> level of flexibility: running multiple hosts with nfsroot and thus
>> shared /etc/fstab tends to be annoying.
>>
>>>> See https://fedoraproject.org/wiki/Features/UsrMove for a more complete
>>>> discussion.
>>>
>>> That does not address the problems of supporting this configuration in a
>>> rolling release.
>>>
>>> Formats in /etc can fall out of sync with software in /usr. If boot
>>> options change, the stuff in /etc/init.d is not updated. If you add
>>> software, the update to /etc/init.d is omitted. If you have a baselayout
>>> change, it is not propagated.
>>
>> Ever heard of CONFIG_PROTECT ? :) What you describe is already what
>> happens and what most people want.
> 
> Leveraging the /usr merge to enable easier updating of multiple systems
> means that you are updating a Gentoo system image on a build server,
> snapshotting /usr both before/after the update and then distributing the
> delta on /usr to other systems without any of the changes that occurred
> outside of /usr. A proper update requires finding all of those changes
> and then applying them manually. That really is not the same thing that
> RHEL and Solaris have, where the necessity of propagating changes
> outside of /usr is minimized by having none to propagate.

After thinking about it some more, maybe git can be (ab)used to do this
by having the image's root be a git repository with /usr in .gitingore.

When updates are done, you would run etc-update on the build host by
committing the changes to / into git with ./usr in .gitignore. Then you
would have the delta from /usr via whatever mechanism that the user
wishes to use with a patch on / from the build system that can be merged
into each target's / repository. The procedure would require more effort
than what Solaris and RHEL do, but if documented, it should fix the
partial update problem that occurs when you do a /usr merge in a rolling
release and then do what people on Solaris and RHEL do to update systems
configured for updates via deltas of /usr.

To give an example, lets assume:

1. /path/to/build/image has a git repository with ./usr in .gitignore on
the build host while the targets have the same (with paths being / and
/usr of course).

2. Things are on ZFS on the build host and the targets.

3. There is a snapshot of the build environment that the targets have.

4. tank/BUILD/gentoo is the / and tank/BUILD/gentoo/usr is the /usr on
the build host.

5. The targets have rpool/SYSTEM/USR/gentoo as their /usr and
rpool/SYSTEM/USR is set readonly (so /usr is not modified due to
inheritance of readonly).

The delta generation would be something like this:

# Setup build environment
sudo -i /path/to/enter-container.sh /path/to/build/image
https://gist.github.com/ryao/3c345f206b19c9795109)

# Update portage
emerge-webrsync / emerge-delta-webrsync / emerge --sync
# Preferably one of the first two with PORTAGE_GPG_* configured

# Install updates
emerge -avDuN @world

# Update config files
etc-update

# Exit build environment
exit

# Commit changes to /
git -C /path/to/build/image commit -a

# Snapshot the build environment
sudo -i zfs snapshot -r tank/BUILD/gentoo@"$(date +%Y%m%d)"

Then a child could be updated by something like:

# First /usr
zfs send -i tank/BUILD/gentoo@previous tank/BUILD/gentoo/usr@"$(date
+%Y%m%d)" | ssh root@$CHILD zfs recv rpool/SYSTEM/USR/gentoo

# Then /
git -C /path/to/build/image diff HEAD^ | ssh root@$CHILD git -C
/other/location apply

# If conflicts occurred, fix whatever was broken
ssh root@$CHILD

# Reboot / restart services
reboot

This is intended to only be an example, but there are a few problems
with this simple example that are worth mentioning:

1. You probably want to have a shell into the system in case the update
to / does not go well, which makes the update to / somewhat hackish.

2. The way Solaris does things is to have boot environments where the
change is in a different boot environment that only takes effect as part
of a reboot. If you are doing a boot environment type thing, you could
probably update / for the new reboot, although you would want to
implement easy rollback if anything goes wrong.

That step on / is somewhat hackish, 

Re: [gentoo-dev] usr merge

[Richard Yao]

On 04/06/2016 12:20 PM, Alexis Ballier wrote:
> On Wednesday, April 6, 2016 6:06:35 PM CEST, Richard Yao wrote:
> 
>> That is unless you put per-system state in /usr/local, do symlinks to it
>> in / and mount /usr/local as part of system boot, which is the other way
>> of doing this. I have seen a variant of this done in asuswrt-merlin on
>> routers.
> 
> This doesnt seem to have anything to do with what I was describing.
> 
> Another option I'm using a lot is nfsroot. This doesn't have the same
> level of flexibility: running multiple hosts with nfsroot and thus
> shared /etc/fstab tends to be annoying.
> 
>>> See https://fedoraproject.org/wiki/Features/UsrMove for a more complete
>>> discussion.
>>
>> That does not address the problems of supporting this configuration in a
>> rolling release.
>>
>> Formats in /etc can fall out of sync with software in /usr. If boot
>> options change, the stuff in /etc/init.d is not updated. If you add
>> software, the update to /etc/init.d is omitted. If you have a baselayout
>> change, it is not propagated.
> 
> Ever heard of CONFIG_PROTECT ? :) What you describe is already what
> happens and what most people want.

Leveraging the /usr merge to enable easier updating of multiple systems
means that you are updating a Gentoo system image on a build server,
snapshotting /usr both before/after the update and then distributing the
delta on /usr to other systems without any of the changes that occurred
outside of /usr. A proper update requires finding all of those changes
and then applying them manually. That really is not the same thing that
RHEL and Solaris have, where the necessity of propagating changes
outside of /usr is minimized by having none to propagate.

I do not understand how CONFIG_PROTECT is relevant here. Whatever
CONFIG_PROTECT did was done on the build system. The systems receiving
the updates via ZFS send/recv or some similar mechanism are not going to
have CONFIG_PROTECT evaluated. Even if it were somehow evaluated, all of
the paths in CONFIG_PROTECT should be outside of /usr anyway.

Re: [gentoo-dev] usr merge

[Richard Yao]

On 04/06/2016 12:06 PM, Richard Yao wrote:
> On 04/06/2016 11:11 AM, Alexis Ballier wrote:
>> On Wednesday, April 6, 2016 4:58:05 PM CEST, M. J. Everitt wrote:
>>> What, if any, is the benefit of squashing /usr out of the equation? I
>>> happen to have a few workstations that load their /usr off an NFS share
>>> presently,
>>
>>
>> This is precisely one case where I see benefits: no need to correlate /
>> and /usr.
>>
>> With the current way, this setup is broken if you don't pay attention:
>> glibc is not backwards compatible (that is, stuff built for glibc 2.22
>> is not guaranteed to work with 2.21 and less), but you have glibc in
>> /lib and stuff in /usr linking and dynamically loading it. If your nfs
>> server updates glibc, you have to update every / on every of your
>> workstations or fear the consequences of running binaries built for 2.22
>> but running against an older version.
>>
> 
> That is unless you put per-system state in /usr/local, do symlinks to it
> in / and mount /usr/local as part of system boot, which is the other way
> of doing this. I have seen a variant of this done in asuswrt-merlin on
> routers.
> 
>> See https://fedoraproject.org/wiki/Features/UsrMove for a more complete
>> discussion.
> 
> That does not address the problems of supporting this configuration in a
> rolling release.
> 
> Formats in /etc can fall out of sync with software in /usr. If boot
> options change, the stuff in /etc/init.d is not updated. If you add
> software, the update to /etc/init.d is omitted. If you have a baselayout
> change, it is not propagated. Whether or not the package manager can be
> used is not discussed. It definitely can be in Solaris when this feature
> is used in Solaris zones, although I am not sure how that interacts with
> updates as I never looked. I do not have a VM with a member of the
> OpenSolaris family handy to check.
> 
> Solaris and RHEL will see the benefits described on the Fedora page
> because they handled many of those problems. In most cases, they handled
> it by being stale non-rolling releases that do not support major version
> upgrades. Fedora handled it by having a disclaimer that things should be
> expected to break across Fedora versions. Neither are things that I
> expect us to do, so if we adopt this, we will need to do something
> entirely new to be able to gain these benefits.

To say it clearly, lets not claim that the /usr merge will give us any
of the benefits mentioned in the Fedora wiki unless we have a plan to
handle the complications that being a rolling distribution poses for
doing atomic updates via the mechanism invented in Solaris on a
post-/usr merge system.

On a non-rolling system release like Solaris or RHEL, you  need to
install information on how to boot daemons with default configurations
in /usr and let users override things in /etc in addition to the /usr
merge to gain the capabilities cited on the Fedora wiki page. You get
bonus points if a clone of a snapshot can be used to make a container
with a working package manager, but could otherwise define that
configuration as being unsupported or write a script to install it.

On a rolling release like Gentoo, rearranging the system baselayout is
also insufficient, but there are many more problems than those that
occur on a non-rolling system release. I listed most of those in my
previous email.

Re: [gentoo-dev] usr merge

[Richard Yao]

On 04/06/2016 11:11 AM, Alexis Ballier wrote:
> On Wednesday, April 6, 2016 4:58:05 PM CEST, M. J. Everitt wrote:
>> What, if any, is the benefit of squashing /usr out of the equation? I
>> happen to have a few workstations that load their /usr off an NFS share
>> presently,
> 
> 
> This is precisely one case where I see benefits: no need to correlate /
> and /usr.
> 
> With the current way, this setup is broken if you don't pay attention:
> glibc is not backwards compatible (that is, stuff built for glibc 2.22
> is not guaranteed to work with 2.21 and less), but you have glibc in
> /lib and stuff in /usr linking and dynamically loading it. If your nfs
> server updates glibc, you have to update every / on every of your
> workstations or fear the consequences of running binaries built for 2.22
> but running against an older version.
> 

That is unless you put per-system state in /usr/local, do symlinks to it
in / and mount /usr/local as part of system boot, which is the other way
of doing this. I have seen a variant of this done in asuswrt-merlin on
routers.

> See https://fedoraproject.org/wiki/Features/UsrMove for a more complete
> discussion.

That does not address the problems of supporting this configuration in a
rolling release.

Formats in /etc can fall out of sync with software in /usr. If boot
options change, the stuff in /etc/init.d is not updated. If you add
software, the update to /etc/init.d is omitted. If you have a baselayout
change, it is not propagated. Whether or not the package manager can be
used is not discussed. It definitely can be in Solaris when this feature
is used in Solaris zones, although I am not sure how that interacts with
updates as I never looked. I do not have a VM with a member of the
OpenSolaris family handy to check.

Solaris and RHEL will see the benefits described on the Fedora page
because they handled many of those problems. In most cases, they handled
it by being stale non-rolling releases that do not support major version
upgrades. Fedora handled it by having a disclaimer that things should be
expected to break across Fedora versions. Neither are things that I
expect us to do, so if we adopt this, we will need to do something
entirely new to be able to gain these benefits.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] usr merge

[Richard Yao]

On 04/06/2016 10:58 AM, M. J. Everitt wrote:
> What, if any, is the benefit of squashing /usr out of the equation? I
> happen to have a few workstations that load their /usr off an NFS share
> presently, with some bodgery-workarounds I did pre the udev notification
> about initramfs's which I have never got around to implementing
> (although I'm pretty sure I have the tools now to do, along with UUIDs
> for boot media).

The idea in Solaris is to enable atomic updates via the /usr mount
without touching data files in /etc or temporary files in /var. Usually,
this would be done on reboot and could be propagated to many systems
either via /usr on NFS or ZFS send/recv.

This works well on Solaris because both software versions are pegged
(such that file formats in /etc are stable) in favor of backported fixes
and the FHS does not change across major OS versions. The same goes for
RHEL.

Gentoo systems managed this way will suffer from multiple problems:

* Software updates that change the configuration file format without
supporting the older format will break.

* Software updates that change the boot scripts will break.

* Future baselayout updates will not be able to touch anything outside
of /usr and anything requiring such things be touched will break.

* An update to /usr that adds new software will fail to include things
outside of /usr, like the boot scripts and configuration files.

* The package database will fall out of sync with /usr (or be broken
period). Presumably, if you are updating this way, you should expect the
package database to be broken.

These are likely to be mostly fixable, but I do not think we have a plan
in place to fix them right now. The general staleness of Solaris and
RHEL handle the first 3 issues for them for free.

I have not looked at the specifics of how Solaris handles the 4th, but I
know that SMF in OpenSolaris descendents will update manifests on first
boot into a new boot environment. That suggests to me that the Solaris
boot scripts handle it by comparing /etc with /usr.

As for the 5th, the package database is not broken in Solaris zones
where the /usr merge is leveraged to enable easy updates. However, I do
not know how updating all zones works when zones have independently
installed software. It might be that the software is installed in
/usr/local inside the zone and conflicts are the user's problem, but it
has been so long since I used an illumos distribution (which is
descended from OpenSolaris) that I do not remember.

> Whilst these aren't currently scheduled for upgrade, I don't personally
> see any merit, given discussions here about work needed to 'shore up' a
> change to match some particular use case. I would therefore definitely
> agree with those that have proposed that this is an Option and not a
> standard gentoo install item unless there are some specific caveats that
> this solves.

The original purpose of the /usr merge in Solaris was to make managing
updates easier. Redhat realized that and copied it. Copying it too
without doing the enabling work necessary for a rolling distribution
would be setting a trap for users who would think that they can manage
deployments of Gentoo like they can manage deployments Solaris and/or RHEL.

> Michael/veremit.
> 
> On 05/04/16 02:19, William Hubbs wrote:
>> All,
>>
>> I thought that since the usr merge is coming up again, and since I lost
>> track of the message where it was brought up, I would open a
>> new thread to discuss it.
>>
>> When it came up before, some were saying that the /usr merge violates
>> the fhs. I don't remember the specifics of what the claim was at the
>> time, (I'm sure someone will point it out if it is still a concern).
>>
>> I don't think creating usr merged stages would be that difficult. I
>> think it would just be a matter of creating a new version of baselayout
>> that puts these symlinks in place:
>>
>> /bin->usr/bin
>> /lib->usr/lib
>> /lib32->usr/lib32
>> /lib64->usr/lib64
>> /sbin->usr/bin
>> /usr/sbin->bin
>>
>> Once that is in place in a new baselayout, I think portage's colission
>> detection would be able to catch files that had the same names and were
>> originally in different paths when building the new stages.
>>
>> I put some thought also in how to nigrate live systems, and I'm not sure
>> what the best way to do that is. I wrote a script, which would do it in
>> theory, but I haven't tested because I only have one system and if
>> it breaks it the only way back would be to reinstall.
>>
>> The script is attached.
>>
>>
>> Thoughts on any of this?
>>
>> William
>>
> 
> 

Re: [gentoo-dev] Re: usr merge

[Richard Yao]

On 04/06/2016 01:34 AM, Duncan wrote:
> Richard Yao posted on Wed, 06 Apr 2016 00:15:58 -0400 as excerpted:
> 
> 
>>> On Apr 4, 2016, at 9:19 PM, William Hubbs <willi...@gentoo.org> wrote:
>>>
>>> All,
>>>
>>> I thought that since the usr merge is coming up again, and since I lost
>>> track of the message where it was brought up, I would open a new thread
>>> to discuss it.
>>>
>>> When it came up before, some were saying that the /usr merge violates
>>> the fhs. I don't remember the specifics of what the claim was at the
>>> time, (I'm sure someone will point it out if it is still a concern).
>>
>> Here are the violations:
>>
>> http://refspecs.linuxfoundation.org/FHS_3.0/
> fhs-3.0.html#binEssentialUserCommandBinaries
>>
>> http://refspecs.linuxfoundation.org/FHS_3.0/
> fhs-3.0.html#sbinSystemBinaries
>>
>> http://refspecs.linuxfoundation.org/FHS_3.0/
> fhs-3.0.html#libEssentialSharedLibrariesAndKern
> 
> (Those links are wrapped and I'm not bothering to jump thru the hoops to 
> unwrap them, since readers can either unwrap them manually or refer to 
> the parent post I'm quoting for the unwrapped versions.)
> 
> If those are the "violations", then putting everything in /usr and making 
> the /bin and /sbin locations symlinks isn't going to be a problem, since 
> /bin and /sbin are specifically allowed to contain symlinks to the 
> executables, instead of the executables themselves, and if the dirs 
> themselves are symlinks to the locations in /usr with the files, that 
> fulfills that requirement.

Alexis Ballier pointed out that the following had been added to make
this okay:

> The following directories, or symbolic links to directories, are
required in /.

As far as the Single UNIX Specification is concerned, this sort of thing
is okay. The Linux Filesystem Hierarchy Standard that further restricts
things had not been updated to permit this until mid-2015.

> And the requirement for /lib is rather vague, saying only that it 
> contains the libs linked by the executables in /bin and /sbin.  So once /
> bin and /sbin are symlinks to the dirs with the executables, /lib (or the 
> arch-specific alternative libdirs) can be a symlink as well.
> 
> Tho I must say doing the reverse, making either /usr itself or /usr/bin 
> and /usr/sbin symlinks to the root dirs, as I did here, actually makes 
> more sense and bends the rules less.

I agree, but Solaris picked the other way and RHEL followed. My opinion
is that we should support both. Any automated upgrade scripts that do
the /usr merge on behalf of users should detect this state and leave it
alone.

That being said, I am not of the opinion that we should migrate existing
installs or drop support for the older layout for a couple reasons. One
is that existing installs will see no benefit from this layout change.
The other is that supporting the existing installs is necessary so that
we can still do QA on them.

I suspect breakage will be more likely with the older layout than the
newer one. If we end up not migrating existing installs automatically
(maybe via a profile change), we might be able to get away with doing
the majority of testing with the older one. If we do not want to do that
long term, we could always do it for something like a year to minimize
the effect on the userbase.

A portage QA check could be made to detect the presence of files in both
/{bin,sbin,lib,lib64,lib32} and /usr/{bin,sbin,lib,lib64,lib32} to make
verifying that packaging does not do anything weird that would
negatively affect the newer layout easier when testing with the older
layout.

The only exception being the linker scripts. There are at least a few of
them still around:

richard@desktop ~ $ for fA in /lib/*; do
> fB=/usr/lib/${fA##*/};
> [[ -f $fA && -f $fB ]] && echo "$fB";
> done
/usr/lib/libeinfo.so
/usr/lib/libpamc.so
/usr/lib/libpam_misc.so
/usr/lib/libpam.so
/usr/lib/librc.so
richard@desktop ~ $ cat /usr/lib/librc.so
/* GNU ld script
   Since Gentoo has critical dynamic libraries in /lib, and the static
versions
   in /usr/lib, we need to have a "fake" dynamic lib in /usr/lib,
otherwise we
   run into linking problems.  This "fake" dynamic lib is a linker
script that
   redirects the linker to the real lib.  And yes, this works in the cross-
   compiling scenario as the sysroot-ed linker will prepend the real path.

   See bug https://bugs.gentoo.org/4411 for more info.
 */
OUTPUT_FORMAT ( elf64-x86-64 )
GROUP ( /lib64/librc.so )

I am a bit out of date on the status of these. Do we still need them? If
we have not already, we could patch GNU ld to eliminate the need for this.

> Basically, what the FHS says, at least in the 3.0 version you linked, is 
> that the e

Re: [gentoo-dev] usr merge

[Richard Yao]

> On Apr 6, 2016, at 4:55 AM, James Le Cuirot  wrote:
> 
> On Wed, 06 Apr 2016 09:42:04 +0200
> Alexis Ballier  wrote:
> 
>>> This was invented in Solaris and copied by RHEL. The upgrade 
>>> path for the /usr merge on those systems is a complete 
>>> reinstall. Upgrading from RHEL6 to RHEL7 this Solaris 10 to 
>>> Solaris 11 is not supported. The reason being that there are 
>>> ways of configuring the system boot process with the original 
>>> layout that break if you try using scripts to migrate to the new 
>>> one. A USE flag for the /usr merge that is off by default would 
>>> allow us to have both worlds without putting any systems at 
>>> risk.  
>> 
>> that's what i'm actually more worried about: the fact they failed to
>> have a proper upgrade path doesnt mean it is impossible, just that it
>> is not easy.
> 
> What about Fedora? This system I'm on now started as Fedora 16 and has
> been upgraded step by step to 23. /bin, /lib, /lib64, and /sbin are
> symlinks but I'm pretty sure it didn't start out that way. I knew the
> change was coming but when it actually happened, I didn't notice for
> quite a while.

The common case can be done by automated scripts. The case where the system is 
configured to mount /usr after init has started is not. In Fedora's case, they 
tell users to expect upgrades to break things, so the people bitten by it at 
least had been warned of breakage before they installed Fedora.
> 
> -- 
> James Le Cuirot (chewi)
> Gentoo Linux Developer
> 

Re: [gentoo-dev] usr merge

[Richard Yao]

> On Apr 4, 2016, at 9:19 PM, William Hubbs  wrote:
> 
> All,
> 
> I thought that since the usr merge is coming up again, and since I lost
> track of the message where it was brought up, I would open a
> new thread to discuss it.
> 
> When it came up before, some were saying that the /usr merge violates
> the fhs. I don't remember the specifics of what the claim was at the
> time, (I'm sure someone will point it out if it is still a concern).

Here are the violations:

http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#binEssentialUserCommandBinaries

http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#sbinSystemBinaries

http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#libEssentialSharedLibrariesAndKern

> 
> I don't think creating usr merged stages would be that difficult. I
> think it would just be a matter of creating a new version of baselayout
> that puts these symlinks in place:
> 
> /bin->usr/bin
> /lib->usr/lib
> /lib32->usr/lib32
> /lib64->usr/lib64
> /sbin->usr/bin
> /usr/sbin->bin
> 
> Once that is in place in a new baselayout, I think portage's colission
> detection would be able to catch files that had the same names and were
> originally in different paths when building the new stages.

We will have users whose system configurations rely on the FHS complain about 
us breaking boot if we force this.

> I put some thought also in how to nigrate live systems, and I'm not sure
> what the best way to do that is. I wrote a script, which would do it in
> theory, but I haven't tested because I only have one system and if
> it breaks it the only way back would be to reinstall.
> 
> The script is attached.
> 
> 
> Thoughts on any of this?


This was invented in Solaris and copied by RHEL. The upgrade path for the /usr 
merge on those systems is a complete reinstall. Upgrading from RHEL6 to RHEL7 
this Solaris 10 to Solaris 11 is not supported. The reason being that there are 
ways of configuring the system boot process with the original layout that break 
if you try using scripts to migrate to the new one. A USE flag for the /usr 
merge that is off by default would allow us to have both worlds without putting 
any systems at risk.

This has been an almost annual debate. I do not have much incentive to keep up 
with it. The reason I ever bothered to explain why this is a bad idea for 
Gentoo was that I was concerned for our user base. My systems would not be 
negatively affected and arguing against changes that originated in Solaris is 
awkward for me.

If others are not willing to be advocates for those users that would only make 
themselves known after an a fundamental change has been made and people are 
determined to go ahead with this, I suggest having and testing a plan for 
backing out the change should the backlash from users after systems break be 
more than people can stomach. This is not the sort of change we should make 
without an "exit strategy".

> William
> 
> 

Re: [gentoo-dev] News item: upgrading to Plasma 5

[Richard Yao]

On Mon, Apr 04, 2016 at 03:34:07AM +1000, Michael Palimaka wrote:
> Hi,
> 
> KDE team intends to stabilise Plasma 5 shortly, so please review the
> accompanying news items.
> 
> Regards,
> 
> Michael
> 
> Title: KDE Plasma 5 Upgrade
> Author: Michael Palimaka 
> Content-Type: text/plain
> Posted: 2016-04-02
> Revision: 1
> News-Item-Format: 1.0
> Display-If-Installed: kde-base/plasma-workspace:4
> 
> KDE Workspaces 4.11 has reached end of life and is no longer supported
> by upstream. It is therefore recommended for all users to upgrade to
> KDE Plasma 5.

Did you mean 4.14? 4.11 should have been EOL a long time ago.

> A detailed upgrade guide is available[1], but in most cases it enough to
> switch to the new desktop/plasma profile, update @world, and
> emerge kde-plasma/plasma-meta:
> 
> # eselect profile list
> # eselect profile set 
> # emerge --ask --changed-use --newrepo --deep world
> # emerge --ask --verbose kde-plasma/plasma-meta
> 
> If you normally use KDM to launch Plasma, note that it is no longer
> supported.
> Upstream recommends x11-misc/sddm instead which is pulled in by
> plasma-meta by
> default. To switch, edit /etc/conf.d/xdm and update DISPLAYMANAGER.
> 
> Due to an an evolution of KDE upstream's release process[2], the traditional
> monolithic KDE 4 release is now split into three distinct components. This
> means that KDE Applications are now separate from the Plasma desktop and
> older KDE 4-based applications will continue to function as normal
> inside Plasma 5.
> 
> KDE Workspaces 4.11 will remain in the tree for a reasonable time, but
> be warned that it is unmaintained and may cause conflicts with
> newer versions of KDE Applications.
> 
> [1] https://wiki.gentoo.org/wiki/KDE/Plasma_5_upgrade
> [2] https://dot.kde.org/2013/09/04/kde-release-structure-evolves
> 
> 

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Richard Yao]

On 02/17/2016 02:01 PM, Andrew Savchenko wrote:
> On Tue, 16 Feb 2016 15:18:46 -0500 Rich Freeman wrote:
>> On Tue, Feb 16, 2016 at 2:31 PM, Patrick Lauer  wrote:
>>>
>>> The failure message comes from rc-mount.sh when the list of PIDs using a
>>> mountpoint includes "$$" which is shell shorthand for self. How can the
>>> current shell claim to be using /usr when it is a shell that only has
>>> dependencies in $LIBDIR ?
>>> As far as I can tell the code at this point calls fuser -k ${list of
>>> pids}, and fuser outputs all PIDs that still use it. I don't see how $$
>>> can end up in there ...
>>
>> What does openrc do when the script fails?  Just shut down the system anyway?
>>
>> If you're going to shut down the system anyway then I'd just force the
>> read-only mount even if it is in use.  That will cause less risk of
>> data loss than leaving it read-write.
>>
>> Of course, it would be better still to kill anything that could
>> potentially be writing to it.
> 
> This is not always possible. Two practical cases from my experience:
> 
> 1) NFS v4 shares can't be unmounted if server is unreachable (even
> with -f). If filesystem (e.g. /home or /) contains such unmounted
> mount points, it can't be unmounted as well, because it is still in
> use. This happens quite often if both NFS server and client are
> running from UPS on low power event (when AC power failed and
> battery is almost empty).

Does `umount -l /path/to/mnt` work on those?

> 2) LUKS device is in frozen state. I use this as a security
> precaution if LUKS fails to unmount (or it takes too long), e.g.
> due to dead mount point.

This gives me another reason to justify being a fan of integrating
encryption directly into a filesystem or using ecryptfs on top of the
VFS. The others were possible integrity concerns (which definitely
happen with a frozen state, although mine were about excessive layering
adding opportunities for bugs) and performance concerns from doing
unnecessary calculations on filesystems that span multiple disks (e.g.
each mirror member gets encrypted independently).

> As far as I understand, mount-ro may be useful only if unmount
> failed, but from my practical experience, openrc just hangs forever
> in such case until UPS is shut down by battery drain.

It is useful even if mounting everything succeeds because of /. That
said, I do not think we can sanely handle every possible configuration
because someone will always come up with something new.

> Best regards,
> Andrew Savchenko
> 




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: rfc: Does OpenRC really need mount-ro

[Richard Yao]

On 02/17/2016 01:32 PM, Rich Freeman wrote:
> On Wed, Feb 17, 2016 at 1:06 PM, Ian Stakenvicius  wrote:
>>
>> Genkernel's initramfs generation was what we endorsed for the most
>> part, until dracut came around.  it's hard to say what "most" are
>> doing but i expect dracut and genkernel based initramfs's make up
>> the vast majority in use by gentoo users, with a small minority
>> rolling their own through other means.
>>
> 
> While I personally endorse dracut over genkernel, the reality is that
> only genkernel is actually documented in the handbook.  This is due at
> least in part to laziness on my part as I've been meaning to add it
> since forever.
> 
> Likewise I intend to update the handbook to make selection of
> openrc/systemd less convoluted as well.  The current handbook does
> offer systemd as an option but then basically refers you out to
> another page that doesn't follow the same flow as the handbook.
> 
> In my notes I've found that it is a pretty trivial change to pick one
> or the other actually if you do it at the right time, so this could be
> added to the handbook with very little disruption to the flow for
> non-systemd users.  I imagine other service managers would be similar,
> or even simpler.  I found that switching between the two only requires
> two changes - one is to pick a systemd profile relatively early in the
> process before doing a world update, and then changing one line in
> your grub config at the end.  If you emerge world after you do most of
> your system configuration systemd will automatically pick up all the
> openrc configuration and use it, which as a bonus leaves you with a
> system that is easy to boot in either mode.
> 
> Getting back to dracut - it is really just a few lines added as an
> alternative to the initramfs section.  After you build your kernel it
> is really just a one-liner, and grub2-mkconfig picks up on it
> automatically (as I imagine it probably does with genkernel as well).
> Unless you want to play with the configuration there isn't much fuss.

dracut does not assist those who do not want generic kernel
configurations. Unfortunately, the handbook does not do a good job in
saying that the initramfs generation and generic kernel configurations
are optional.

> I think we really should give strong consideration to recommending
> dracut as a default, while of course preserving the option of
> genkernel.  I'm certainly open to feedback if there is some use case
> where genkernel is better, but dracut is cross-distro, gives you
> options to easily maximize or minimize your config, and is really easy
> to tailor with modules.

There is no default and system boot without an initramfs not only works,
but is advisable for faster boot unless something fancy is being done
that needs it.

Claiming to pick a default between genkernel and dracut when both are
optional makes no sense, especially since dracut's capabilities
(initramfs generation) are a subset of genkernel's (initramfs generation
and kernel builds). dracut could replace genkernel's initramfs
generation capabilities, but it simply cannot replace genkernel for
building a generic kernel. It was never intended to do that.

By the way, pver the course of time, there have been things genkernel
did better and things dracut did better. It is unlikely one will ever be
superior to the other. However, some feedback on what genkernel does
poorly versus dracut and could therefore improve would be helpful.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: rfc: Does OpenRC really need mount-ro

[Richard Yao]

On 02/17/2016 12:19 PM, Rich Freeman wrote:
> On Wed, Feb 17, 2016 at 9:24 AM, Richard Yao <r...@gentoo.org> wrote:
>> Systemd installs that go back into the initramfs at shutdown are rare 
>> because there is a
>> hook for the initramfs to tell systemd that it should re-exec it and very 
>> few configurations
>> do that. Even fewer that do it actually need it.
> 
> While I won't debate that it probably isn't strictly essential, dracut
> handles unmounting root for systemd just fine (well, at least on
> non-nfs - the version I'm using with an nfs root struggles in this
> regard, though unclean shutdown on nfs with no files open probably
> isn't really a problem).

Dracut handling it well is not up for dispute. When I checked last year,
dracut simply did not tell systemd to use this functionality because it
was unnecessary functionality that only served to slowed down the
shutdown process. It only enables it when a driver indicates an actual
need, which is the way that it should be.

> Is dracut still not widely used?  I know that it was all the fashion
> for a decade or two for every distro to build their own initramfs, but
> I don't get why anybody wouldn't just make the switch - it is far more
> capable and configurable.

Not many Gentoo users use dracut. It does not handle kernel compilation
or bootloader configuration. It is definite ahead of genkernel in
networking though, but there is not much demand for that among users.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

On 02/17/2016 01:47 PM, Rich Freeman wrote:
> On Wed, Feb 17, 2016 at 11:04 AM, Richard Yao <r...@gentoo.org> wrote:
>>
>> This is something that I think many of us who had systems broken by
>> sys-fs/udev multiple times before sys-fs/eudev was an option thought was
>> obvious.
> 
> About the only "system-breaking" change I'm aware of in udev over the
> years was the change in default network interface names.  That was
> preceded by news on how to avoid the change, or how to adapt to the
> change.
> 
> There certainly wasn't some change introduced without warning that
> just broke systems in random ways.
> 
>> If a complete list of the breakages that lead to the creation of
>> sys-fs/eudev were produced, I imagine that the list would have at least
>> 3 to 5 items from the ~18 months before sys-fs/eudev with half of them
>> were probably self inflicted by sys-fs/udev maintenance.
> 
> Anytime upstream changes something it is up to package maintainers to
> evaluate the change and adapt to it as needed, especially for critical
> packages like udev.  For the most part I think this is happening.
> Whether it is the udev maintainers doing the QA or the eudev
> maintainers doing the QA, somebody has to do the QA (and in Gentoo it
> sounds like we do it twice, which is fine).
> 
>> I recall one incident involving whether udev should be in /sbin or
>> /usr/sbin being resolved after 6 months of debate between then future
>> eudev founders and sys-fs/udev maintainers only because the systemd
>> developers told the sys-fs/udev maintainers it should be in /sbin like
>> others had told them.
> 
> So, this sounds like a disagreement between the future eudev founders
> and the udev maintainers in Gentoo.  It really has nothing to do with
> udev itself.

That is just one thing that I remembered off the top of my head and
quite frankly, that situation was the most absurd system breakage
incident that ever occurred involving udev. The arguments by the
sys-fs/udev maintainers at the time were that upstream wanted it that
way when even the systemd developers did not agree with them. The matter
was only resolved after one of the sys-fs/udev maintainers decided to
ask the systemd developers what they actually thought after 6 months of
claiming their way was the upstream way and were told that they thought
the packaging was wrong. Everyone else had believed the sys-fs/udev
maintainers' statements that upstream actually thought such things, and
consequently, were adamant that the systemd maintainers had no idea what
they were doing.

There were multiple breakages caused by sys-fs/udev maintainance
following systemd assimilating udev based on the principle that upstream
wanted it that way. The outsourcing of responsibility for thought on
such things were one of the things that contributed to eudev's creation.
Another were absolute refusal by Kay Sievers at the time to fix
regressions when given patches. It was not "rewrite the patch this way".
It was along the lines of "we are not fixing that because we don't care
about people affected by this and fixing that would add 40 lines to the
codebase".

> And that is OK - we're allowed to have the same package maintained
> under two different names by two sets of maintainers.  Obviously it
> isn't ideal, and it would be better if everybody could agree.
> 
>> Another broke support for older kernels for no apparent benefit (and
>> this sort of regression naturally enters sys-fs/udev):
> 
> That isn't really the same as "breaking Gentoo."  And as was pointed
> out they did accept patches to provide support back.

That is a new development.

> I think it is a bit unfair to characterize the udev maintainers as
> breaking things left and right.  They apparently differ with you in
> how they prefer to set their defaults, and what their dependencies
> are.  They apparently also accept patches when you provide them for
> older kernels, which is what upstreams are supposed to do.

There were multiple incidents where either the sys-fs/udev maintainers
or the systemd developers refused to listen to reason. Systems broke
because of it and there were no warnings.

> It really seems like the main reasons for eudev's existence right now
> are (based on this thread):
> 1.  The eudev maintainers don't trust the udev maintainer's QA and
> want to do their own layer of QA before introducing changes to the
> tree.
> 2.  The eudev maintainers prefer a different default network interface
> naming scheme (my understanding is that eudev can be configured to
> behave as udev does by default, and vice-versa - for example, on the
> box I'm typing this on my packets are going out over eth0 just fine on
> systemd).
> 3.  The eudev tarballs are smaller

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

On 02/17/2016 11:16 AM, Michał Górny wrote:
> On Wed, 17 Feb 2016 14:38:05 +0100
> Chí-Thanh Christopher Nguyễn  wrote:
> 
>> Michał Górny schrieb:
 With the exception that Lennart Poettering is the lead developer of
 systemd/udev, while such a thing cannot be said about you and eudev.  
>>> He's lead developer of *systemd*. udev is a split part of systemd
>>> codebase which has specific maintainers.  
>>
>> systemd and udev share the same codebase. You can no longer build udev 
>> without systemd. udev is only a sub-project of systemd now, hence the 
>> name "systemd-udevd".
> 
> Of course, sure. But since you seem not to be able to understand
> basics: this *does not* mean Lennart is the only person having
> influence on how udev progresses. Sharing the same repository
> and utility libraries is not the same as being the same thing.
> 
> Much like the Council does not strictly force the development of eudev.
> 

According to _AxS_, there are very few differences between systemd-udev
and eudev at the present time:

16:14 <@_AxS_> At this point there really aren't much in terms of
differences -- most of the code is the same as upstream, but the file
structure is different and obviously the build system is proper and
standalone.  There's the rule-generator patchset but otherwise
   things are pretty well the same iirc.
16:14 <@_AxS_> We *did* have a bunch of other things in eudev that
improved matters for older kernels or just general code/memory
improvements, but those have since been integrated upstream iirc
16:15 <@ryao> _AxS_: I recall hearing that eudev regressed with respect
to 2.6.32. If the older versions were not good enough and I had time, I
would try fixing that.
16:16 <@ryao> _AxS_: It is interesting to hear that patches to support
older kernels are now being merged by them. Kay told me that such
patches would not be merged.
16:18 <@_AxS_> ryao: eudev-3.x doesn't support the older kernels, true.
 However we determined that was ok because (1) the older kernels like
openvz have had the appropriate functionality backported into them, and
(2) we're keeping the older eudev versions around.

The main differences at this time seem to be the rule-generator patchset
for users who like that, a better build system (what IcedTea is to
OpenJDK) and a stricter process for getting newer code. Specifically,
instead of just doing a revision bump with the implicit assumption that
it was tested, every change is reviewed and committed to a repository
that is tagged independently of systemd-udev.

If there are regressions because of something upstream did, which has
occurred in the past, and they made it into eudev, there is someone who
accepted the commit that is responsible for it. They should not only
have caught it, but they also should have put a plan in place for
dealing with it like was done with the regressed kernel version support.
If they did not, then the eudev project can revise its QA strategies to
ensure that it does not happen again. That is something that can only be
rigorously done with a separate project that imports code from a vendor
like FreeBSD does, which is precisely what eudev is.

That is clearly better than what we have with sys-fs/udev. Given that
the two are almost identical with sys-fs/eudev being better suited for
systems where PID 1 is not systemd, there really is no point in having
sys-fs/udev be the first provider for virtual/udev. There is also no
point in having sys-fs/udev become a separate package from systemd,
unless we decide to break every component of systemd from the part meant
to be PID 1 too, but that is contrary to what systemd upstream recommends.

If making sys-fs/eudev the default virtual/udev provider turns out to be
a bad decision, we can always revisit it, but it seems better than the
status quo that we have now. Having less review of new code is bad and
having systemd-udev split from the systemd package on systems that use
systemd makes no sense. If we continue having the split, systemd would
depend on sys-fs/udev and consequently the systemd profiles with systemd
in @system would be unaffected by whatever the provider order happens to be.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

On 02/17/2016 09:54 AM, brettrse...@gmail.com wrote:
>  
> Sent from my Verizon Wireless BlackBerry
> 
> -Original Message-
> From: Ben Kohler <bkoh...@gmail.com>
> Date: Wed, 17 Feb 2016 08:01:32 
> To: <gentoo-dev@lists.gentoo.org>
> Reply-to: gentoo-dev@lists.gentoo.org
> Subject: Re: [gentoo-dev] Changing order of default virtual/udev provider
> 
> On Wed, Feb 17, 2016 at 7:55 AM, Richard Yao <r...@gentoo.org> wrote:
> 
>>
>>
>> eudev has every commit scrutinized by people who care about using it on
>> Gentoo. systemd-udev does not. Consequently, eudev has avoided the system
>> boot breaking regressions that prompted its creation. That is a good reason
>> to make it the new default. If it fails to fulfill its duties, then this
>> could be revisited, but that should be unlikely.
>>
>> I think if someone could enumerate those specific breakages and present it
> as evidence, that could get more people on board for this change.  Moreso
> than just "upstream doesn't care about us" or "eventually split udev will
> be impossible".
> 
> -Ben
> 

This is something that I think many of us who had systems broken by
sys-fs/udev multiple times before sys-fs/eudev was an option thought was
obvious.

If a complete list of the breakages that lead to the creation of
sys-fs/eudev were produced, I imagine that the list would have at least
3 to 5 items from the ~18 months before sys-fs/eudev with half of them
were probably self inflicted by sys-fs/udev maintenance.

I recall one incident involving whether udev should be in /sbin or
/usr/sbin being resolved after 6 months of debate between then future
eudev founders and sys-fs/udev maintainers only because the systemd
developers told the sys-fs/udev maintainers it should be in /sbin like
others had told them.

Another broke support for older kernels for no apparent benefit (and
this sort of regression naturally enters sys-fs/udev):

https://github.com/gentoo/eudev/commit/eeb8d70a6b38f736febaa4c3b03e39b4c1193a6c

There were other issues too, but I am unable to volunteer the time
required to go through history to figure out what was broken, when and
for how long.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

> On Feb 17, 2016, at 9:41 AM, Richard Yao <r...@gentoo.org> wrote:
> 
> 
>> On Feb 17, 2016, at 9:01 AM, Ben Kohler <bkoh...@gmail.com> wrote:
>> 
>> 
>> 
>>> On Wed, Feb 17, 2016 at 7:55 AM, Richard Yao <r...@gentoo.org> wrote:
>>> 
>>> 
>>> eudev has every commit scrutinized by people who care about using it on 
>>> Gentoo. systemd-udev does not. Consequently, eudev has avoided the system 
>>> boot breaking regressions that prompted its creation. That is a good reason 
>>> to make it the new default. If it fails to fulfill its duties, then this 
>>> could be revisited, but that should be unlikely.
>> I think if someone could enumerate those specific breakages and present it 
>> as evidence, that could get more people on board for this change.  Moreso 
>> than just "upstream doesn't care about us" or "eventually split udev will be 
>> impossible".
> 
> That would require more time than I have to give right now. Hopefully someone 
> else could go through the old bug reports and IRC logs to find those records.

I forgot to mention commit histories. The change log on CVS for sys-fs/udev 
should provide information on things that were broken for long spans of time if 
the causes of the fixes are scrutinized while commits to eudev fixing things 
that were broken (and likely still are) would also document things. At least 
some of the early ones under my name were rejected by systemd. They went into 
eudev when it was created.
> 
>> -Ben

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

> On Feb 17, 2016, at 9:01 AM, Ben Kohler <bkoh...@gmail.com> wrote:
> 
> 
> 
>> On Wed, Feb 17, 2016 at 7:55 AM, Richard Yao <r...@gentoo.org> wrote:
>> 
>> 
>> eudev has every commit scrutinized by people who care about using it on 
>> Gentoo. systemd-udev does not. Consequently, eudev has avoided the system 
>> boot breaking regressions that prompted its creation. That is a good reason 
>> to make it the new default. If it fails to fulfill its duties, then this 
>> could be revisited, but that should be unlikely.
> I think if someone could enumerate those specific breakages and present it as 
> evidence, that could get more people on board for this change.  Moreso than 
> just "upstream doesn't care about us" or "eventually split udev will be 
> impossible".

That would require more time than I have to give right now. Hopefully someone 
else could go through the old bug reports and IRC logs to find those records.

> -Ben

Re: [gentoo-dev] Re: rfc: Does OpenRC really need mount-ro

[Richard Yao]

> On Feb 16, 2016, at 9:20 PM, Duncan <1i5t5.dun...@cox.net> wrote:
> 
> William Hubbs posted on Tue, 16 Feb 2016 12:41:29 -0600 as excerpted:
> 
>> What I'm trying to figure out is, what to do about re-mounting file
>> systems read-only.
>> 
>> How does systemd do this? I didn't find an equivalent of the mount-ro
>> service there.
> 
> For quite some time now, systemd has actually had a mechanism whereby the 
> main systemd process reexecs (with a pivot-root) the initr* systemd and 
> returns control to it during the shutdown process, thereby allowing a 
> more controlled shutdown than traditional init systems because the final 
> stages are actually running from the virtual-filesystem of the initr*, 
> such that after everything running on the main root is shutdown, the main 
> root itself can actually be unmounted, not just mounted read-only, 
> because there is literally nothing running on it any longer.
> 
> There's still a fallback to read-only mounting if an initr* isn't used or 
> if reinvoking the initr* version fails for some reason, but with an 
> initr*, when everything's working properly, while there are still some 
> bits of userspace running, they're no longer actually running off of the 
> main root, so main root can actually be unmounted much like any other 
> filesystem.

Systemd installs that go back into the initramfs at shutdown are rare because 
there is a hook for the initramfs to tell systemd that it should re-exec it and 
very few configurations do that. Even fewer that do it actually need it.

The biggest user of that mechanism of which I am aware is ZFS on EL/Fedora when 
booted with Dracut. It does not need it and it was only implemented was that 
someone who did not understand how ZFS was designed to integrate with the boot 
and startup processes thought it was a good idea.

As it turns out, that behavior actually breaks the mechanism intended to make 
multipath sane by marking the pool in such a way that it tells all systems with 
access to the disks that a pool that will be used on next boot is not going to 
be used by anyone. If they import it and the system boots, the pool can be 
damaged beyond repair.

Thankfully, no one seems to boot EL/Fedora systems off ZFS pools in multipath 
environments. The code to hook into this special behavior will be removed in 
the future, but that is a low priority as none of the developers' employers 
care about it and the almost negligible possibility that the mechanism would 
save someone from data loss  has made it too low of a priority for any of us to 
spend our free time on it.

> The process is explained a bit better in the copious blogposted systemd 
> documentation.  Let's see if I can find a link...
> 
> OK, this isn't where I originally read about it, which IIRC was aimed 
> more at admins, while this is aimed at initr* devs, but that's probably a 
> good thing as it includes more specific detail...
> 
> https://www.freedesktop.org/wiki/Software/systemd/InitrdInterface/
> 
> And here's some more, this time in the storage daemon controlled root and 
> initr* context...
> 
> https://www.freedesktop.org/wiki/Software/systemd/RootStorageDaemons/
> 
> 
> But... all that doesn't answer the original question directly, does it?  
> Where there's no return to initr*, how /does/ systemd handle read-only 
> mounting?
> 
> First, the nice ascii-diagram flow charts in the bootup (7) manpage may 
> be useful, in particular here, the shutdown diagram (tho IDK if you can 
> find such things useful or not??).
> 
> https://www.freedesktop.org/software/systemd/man/bootup.html
> 
> Here's the shutdown diagram described in words:
> 
> Initial shutdown is via two targets (as opposed to specific services), 
> shutdown.target, which conflicts with all (normal) system services 
> thereby shutting them down, and umount.target, which conflicts with file 
> mounts, swaps, cryptsetup device, etc.  Here, we're obviously interested 
> in umount.target.  Then after those two targets are reached, various low 
> level services are run or stopped, in ordered to reach final.target.  
> After final.target, the appropriate systemd-(reboot|poweroff|halt|kexec) 
> service is run, to hit the ultimate (reboot|poweroff|halt|kexec).target, 
> which of course is never actually evaluated, since the service actually 
> does the intended action.
> 
> The primary takeaway is that you might not be finding a specific systemd 
> remount-ro service, because it might be a target, defined in terms of 
> conflicts with mount units, etc, rather than a specific service.
> 
> Neither shutdown.target nor umount.target have any wants or requires by 
> default, but the various normal services and mount units conflict with 
> them, either via default or specifically, so are shut down before the 
> target can be reached.
> 
> final.target has the After=shutdown.target umount.target setting, so 
> won't be reached until they are reached.
> 
> The respective 

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Richard Yao]

> On Feb 16, 2016, at 3:18 PM, Rich Freeman  wrote:
> 
>> On Tue, Feb 16, 2016 at 2:31 PM, Patrick Lauer  wrote:
>> 
>> The failure message comes from rc-mount.sh when the list of PIDs using a
>> mountpoint includes "$$" which is shell shorthand for self. How can the
>> current shell claim to be using /usr when it is a shell that only has
>> dependencies in $LIBDIR ?
>> As far as I can tell the code at this point calls fuser -k ${list of
>> pids}, and fuser outputs all PIDs that still use it. I don't see how $$
>> can end up in there ...
> 
> What does openrc do when the script fails?  Just shut down the system anyway?
> 
> If you're going to shut down the system anyway then I'd just force the
> read-only mount even if it is in use.  That will cause less risk of
> data loss than leaving it read-write.
> 
> Of course, it would be better still to kill anything that could
> potentially be writing to it.

Agreed.
> 
> -- 
> Rich
> 

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Richard Yao]

> On Feb 16, 2016, at 1:41 PM, William Hubbs  wrote:
> 
>> On Tue, Feb 16, 2016 at 01:22:13PM -0500, Rich Freeman wrote:
>>> On Tue, Feb 16, 2016 at 1:05 PM, William Hubbs  wrote:
>>> 
>>> The reason it exists is very vague to me; I think it has something to do
>>> with claims of data loss in the past.
>> 
>> Is there some other event that will cause all filesystems to be
>> remounted read-only or unmounted before shutdown?
> 
> When localmount/netmount stop they try to unmount file systems they know
> about, but they do not try to remount anything.
> 
> 
>> You definitely will want to either unmount or remount readonly all
>> filesystems prior to rebooting.  I don't think the kernel guarantees
>> that this will happen (I'd have to look at it).  Just doing a sync
>> before poweroff doesn't seem ideal - if nothing else it will leave
>> filesystems marked as dirty and likely force fscks on the next boot
>> (or at least it should - if it doesn't that is another opportunity for
>> data loss).
>> 
>> There are different ways of accomplishing this of course, but you
>> really want to have everything read-only in the end.
> 
> unmounting is easy enough; we already do that.
> 
> What I'm trying to figure out is, what to do about re-mounting file
> systems read-only.
> 
> How does systemd do this? I didn't find an equivalent of the mount-ro
> service there.

One idea proposed by systemd that is almost never used in production is to fall 
back to an initramfs environment to undo the boot process by umounting /. It 
would not surprise me if the normal case were hard coded to remount / as ro 
because you risk filesystem corruption otherwise. Journaling filesystems are 
fairly good at surviving that, but you are still taking a risk due to partial 
writes and anyone using ext2 would be taking a much bigger gamble.
> 
> William

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Richard Yao]

> On Feb 16, 2016, at 1:05 PM, William Hubbs  wrote:
> 
> All,
> 
> I have a bug that points out a significant issue with
> /etc/init.d/mount-ro in OpenRC.
> 
> Apparently, there are issues that cause it to not work properly for file
> systems which happen to be pre-mounted from an initramfs [1].
> 
> This service only exists in the Linux world; there is no equivalent in
> OpenRC for any other operating systems we support.
> 
> The reason it exists is very vague to me; I think it has something to do
> with claims of data loss in the past.
> 
> I'm asking for more specific information, and if there is none, due to
> the bug I lincluded in this message, I am considering removing this
> service in 0.21 since I can't find an equivalent anywhere else.

If you shutdown the system while ext4 or XFS is mounted rw, then there is no 
guarantee that dirty data writeout finishes and you can get situations where 
partial writes to disks from power being cut at the end of the shutdown process 
kills the filesystem.

That said, ZFS does not need to be remounted rw because it is an atomic 
filesystem. There might be data loss, but it would only be the last 5 seconds 
of unsynced data (which is safe to use) and there would be no risk of killing 
it.

> Thanks,
> 
> William
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=573760

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

> On Feb 17, 2016, at 8:47 AM, Ben Kohler <bkoh...@gmail.com> wrote:
> 
> 
> 
>> On Wed, Feb 17, 2016 at 7:39 AM, Richard Yao <r...@gentoo.org> wrote:
>> 
>> I have no idea why we are even discussing the choice of default for 
>> virtual/udev to have subdiscussions about kdbus. Practically everyone on the 
>> list thinks eudev is the best choice.
> 
> I think a lot of us appreciate that eudev exists as a lifeboat or backup 
> plan, but want to wait until there's a real, obvious, technical reason to 
> switch.  MOST of the reasons given have just been vague predictions of future 
> doom and gloom.  And if we dare ask for more technical reasons, we get 
> berated for being a "systemd lover".
> 
> "Let's wait until udev becomes unusable" doesn't seem that unreasonable to 
> me, and it has nothing to do with being pro or anti systemd.

eudev has every commit scrutinized by people who care about using it on Gentoo. 
systemd-udev does not. Consequently, eudev has avoided the system boot breaking 
regressions that prompted its creation. That is a good reason to make it the 
new default. If it fails to fulfill its duties, then this could be revisited, 
but that should be unlikely.

> 
> -Ben
> 

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

> On Feb 17, 2016, at 5:52 AM, Alexis Ballier <aball...@gentoo.org> wrote:
> 
> On Tue, 16 Feb 2016 23:00:27 -0500
> Richard Yao <r...@gentoo.org> wrote:
> 
>>> On 02/08/2016 10:09 PM, Rich Freeman wrote:
>>> On Mon, Feb 8, 2016 at 7:58 PM, Anthony G. Basile  
>> <bluen...@gentoo.org> wrote:
>>>> 
>>>> what does in-house tool mean?  i'm a gentoo developer but i also
>>>> work on an upstream project (eudev) that 14 distros use.
>>>> 
>>>> some of the criticism given here are my concerns as well and i've
>>>> spoken with the various distros --- slack, parted magic, puppy.
>>>> they get what's going on and they still see eudev is the best way
>>>> forward for now.  it may not be in the future, but neither will a
>>>> udev extracted from a compiled full systemd codebase.  
>>> 
>>> How many of those 14 distros have more than 14 users?
>>> 
>>> Look, I get it, some people don't like systemd.  That's fine.
>>> However, you have to realize at this point that a non-systemd
>>> configuration is anything but mainstream.  There will always be a
>>> "poppyseed linux" whose purpose in life seems to be to preserve
>>> linux without sysfs or some other obscure practice.  I just think
>>> that Gentoo should offer the choice to do those things, but have a
>>> more mainstream set of defaults.  
>> 
>> The new mainstream is docker. Docker recently switched to Alpine
>> Linux, which uses OpenRC+eudev:
>> 
>> https://www.brianchristner.io/docker-is-moving-to-alpine-linux/
>> 
>> That dwarfs whatever marketshare systemd has in the same way that
>> Android+iOS dwarfed whatever marketshare Windows has.
>> 
>> If userbase is what matters to you, then OpenRC+eudev won. It is the
>> logical choice for those concerned about userbase because that is what
>> the Linux ecosystem will be using going forward.
>> 
>> I do not think userbase should be the sole means by which we make
>> decisions, but those that think otherwise should now join the
>> eudev+OpenRC camp. It has the bigger userbase share going forward.
>> 
>> To put it another way, the war is over. Welcome abroad. :)
> 
> I don't know docker well enough, but an lxc container definitely doesn't
> use udev inside the container.

There really is no point to running udev inside the container, but if you do 
run udev in an Alpine Linux docker container, you get eudev.

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

> On Feb 17, 2016, at 7:58 AM, Michał Górny <mgo...@gentoo.org> wrote:
> 
> On Wed, 17 Feb 2016 07:53:22 -0500
> Richard Yao <r...@gentoo.org> wrote:
> 
>>> On Feb 17, 2016, at 7:43 AM, Michał Górny <mgo...@gentoo.org> wrote:
>>> 
>>> On Tue, 16 Feb 2016 23:41:33 +0100
>>> Chí-Thanh Christopher Nguyễn <chith...@gentoo.org> wrote:
>>> 
>>>> Alexis Ballier schrieb:  
>>>>>>> If it's just that, it's not limited to udev, but anything using
>>>>>>> kdbus/bus1, and would mean openrc/${favorite init system} will have
>>>>>>> to do the same thing anyway. But again, almost 2 years is extremely
>>>>>>> old considering all the flux that has been around kbus.
>>>>>> 
>>>>>> OpenRC itself can for now just ignore kdbus, bus1, or whatever kernel
>>>>>> IPC system comes next.
>>>>> 
>>>>> Well, as Lennart wrote it, kbus would have needed some initialisation.
>>>>> Just like we have a dbus init script, openrc would have a kdbus one.
>>>>> 
>>>>>> But if upstream udev makes use of the systemd
>>>>>> userspace interface to the kernel IPC system, then OpenRC would have
>>>>>> to implement the same interface in order to have working udev.
>>>>> 
>>>>> As I understand it, a kernel IPC doesn't need systemd to work. udev
>>>>> might use wrappers from libsystemd, or libbus1, just like we have
>>>>> programs using libv4l or libbluetooth currently.
>>>> 
>>>> In a follow-up, upstream wrote about how you should only run udev together 
>>>> with systemd, and if you don't want to do that (spelling as in original):
>>>> 
>>>> "we will not support the udev-on-netlink case anymore. I see three options:
>>>> a) fork things, b) live with systemd, c) if hate systemd that much, but
>>>> love udev so much, then implement an alternative userspace for kdbus to
>>>> do initialiuzation/policy/activation."
>>>> https://lists.freedesktop.org/archives/systemd-devel/2014-May/019664.html
>>>> 
>>>> So it seems a bit more than only initialization is needed.  
>>> 
>>> You're missing the third option which is a sane option, and jump
>>> straight to pitchforks.
>>> 
>>> As I see it, *if* this becomes a necessity, we're quite like are going
>>> to provide KDBUS parts of systemd the way we provide udev parts right
>>> now. After all, libsystemd-bus will be useful to more applications.
>>> 
>>> Of course, someone may want to fork that into libebus just for the sake
>>> of renaming.
>>> 
>>> And after all, as it has already been noted, there are people
>>> interested in maintaining non-systemd userspace for KDBUS. Which is
>>> kinda the obvious choice, unlike forking something.  
>> 
>> kdbus is dead. It is fatally flawed and Greg is no longer trying to get it 
>> merged as he is not updating his branch for newer kernel versions. If I 
>> recall correctly, kdbus was also removed from Fedora and has no distribution 
>> backing it anymore.
> 
> Then... why are we even discussing this?

I have no idea why we are even discussing the choice of default for 
virtual/udev to have subdiscussions about kdbus. Practically everyone on the 
list thinks eudev is the best choice. From the perspective of the Linux 
community going forward through docker, eudev is the only sensible choice for a 
udev-based system for those that want to use the same code everyone else uses.

All arguments about systemd are akin to arguments about how Windows 10 is the 
next thing in desktops in a world dominated by POSIX through iOS and Android. 
Coincidentally, neither use systemd.

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

> On Feb 17, 2016, at 1:37 AM, Michał Górny <mgo...@gentoo.org> wrote:
> 
> On Tue, 16 Feb 2016 21:54:31 -0500
> Richard Yao <r...@gentoo.org> wrote:
> 
>>> On 02/08/2016 07:46 AM, Michał Górny wrote:
>>> On Mon, 8 Feb 2016 10:08:22 +0100
>>> Patrick Lauer <patr...@gentoo.org> wrote:
>>> 
>>>> Ohey,
>>>> 
>>>> I've opened a bug at:
>>>> https://bugs.gentoo.org/show_bug.cgi?id=573922
>>>> 
>>>> The idea here is to change the order of the providers of virtual/udev.
>>>> For existing installs this has zero impact.
>>>> For stage3 this would mean that eudev is pulled in instead of udev.
>>>> 
>>>> The rationale behind this is:
>>>> 
>>>> * eudev is an in-house fork, and there's more than a dozen distros
>>>> already using it by default that are not us. Which is a little bit weird 
>>>> ...  
>>> 
>>> That's not an argument. I can also fork random system components. Would
>>> you consider that a reason to replace the defaults with our 'in-house'
>>> forks?
>>> 
>>>> * Both udev and eudev have pretty much feature parity, so there won't be
>>>> any user-visible changes
>>>> 
>>>> * udev upstream strongly discourages standalone udev (without systemd)
>>>> since at least 2012
>>>> 
>>>> (see for example:
>>>> https://lists.freedesktop.org/archives/systemd-devel/2012-June/005516.html
>>>> https://lkml.org/lkml/2012/10/3/618
>>>> )
>>>> 
>>>> So it'd be (1) following upstreams recommendations and (2) dogfooding
>>>> our own tools. I don't see any downsides to this :)  
>>> 
>>> I'm strongly against this, because:
>>> 
>>> 1. It is a conflict-induced fork. As such, it will never be merged
>>> upstream and it will never be supported upstream. In fact, it is
>>> continually forces to follow upstream changes and adapt to them. eudev
>>> is more likely to break because of the Gentoo developer(s) working hard
>>> to merge upstream changes to their incompatible code.  
>> 
>> That was the entire point of the project. Upstream rejected any attempts
>> to do things that we actually needed and broke things claiming the
>> distributions were responsible for handling the breakage, so eudev was
>> started on the basis that we needed a project that would ensure that
>> changes in udev occur in a way that makes sense.
> 
> What things? Things like 'let's try to spawn this script third time in
> case someone mounted something so that it happens to work this time'?
> Yes, that old behavior really made sense.
> 
>>> 2. Many of Gentoo users are programmers who appreciate the 'vanilla'
>>> API experience Gentoo often provides. Switching the defaults to a fork
>>> that is known to intentionally diverge from upstream goes against that
>>> principle. Programs written against eudev may not work correctly with
>>> upstream udev.  
>> 
>> If upstream udev were stable, that would be one thing, but it
>> intentionally diverges from itself continuously. The only experience
>> that could be reliably provided with upstream udev is one of continual
>> breakage.
> 
> This is FUD, without any proof.

That kind of breakage is the reason eudev exists. If it did not occur, I would 
have not organized people to start eudev a few years ago. Part of the 
motivation was inflicted by the sys-fs/udev maintainers at the time, but 
systemd upstream did it's fair share by rejecting compatibility patches on the 
basis that only downstream should handle those. eudev was founded to be that 
downstream and also be upstream as sys-fs/udev refused to take patches that 
systemd would not take.
> 
>> 
>>> 3. eudev has fallen behind systemd/udev more than once in the past,
>>> and caused visible breakage to users this way.  
>> 
>> When?
> 
> Whenever we installed new systemd and udev versions, and needed to bump
> the dependency in virtual, and eudev maintainers were nowhere to be
> found.
> 
>> Can we also consider all of the times udev broke the boot process
>> because upstream just didn't care about doing changes in a sane way and
>> the people interested in providing the upstream experience delivered on
>> that goal?
> 
> Proof needed.

Go back and look at logs of me talking to WilliamH and talking in #systemd in 
the year before eudev was founded. There were problems with separate user among 
others. By the time, eudev was started, it was a wonde

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

> On Feb 17, 2016, at 5:34 AM, Michał Górny <mgo...@gentoo.org> wrote:
> 
> Dnia 17 lutego 2016 05:00:27 CET, Richard Yao <r...@gentoo.org> napisał(a):
>>> On 02/08/2016 10:09 PM, Rich Freeman wrote:
>>> On Mon, Feb 8, 2016 at 7:58 PM, Anthony G. Basile
>> <bluen...@gentoo.org> wrote:
>>>> 
>>>> what does in-house tool mean?  i'm a gentoo developer but i also
>> work
>>>> on an upstream project (eudev) that 14 distros use.
>>>> 
>>>> some of the criticism given here are my concerns as well and i've
>>>> spoken with the various distros --- slack, parted magic, puppy.
>> they
>>>> get what's going on and they still see eudev is the best way forward
>>>> for now.  it may not be in the future, but neither will a udev
>>>> extracted from a compiled full systemd codebase.
>>> 
>>> How many of those 14 distros have more than 14 users?
>>> 
>>> Look, I get it, some people don't like systemd.  That's fine.
>>> However, you have to realize at this point that a non-systemd
>>> configuration is anything but mainstream.  There will always be a
>>> "poppyseed linux" whose purpose in life seems to be to preserve linux
>>> without sysfs or some other obscure practice.  I just think that
>>> Gentoo should offer the choice to do those things, but have a more
>>> mainstream set of defaults.
>> 
>> The new mainstream is docker. Docker recently switched to Alpine Linux,
>> which uses OpenRC+eudev:
>> 
>> https://www.brianchristner.io/docker-is-moving-to-alpine-linux/
>> 
>> That dwarfs whatever marketshare systemd has in the same way that
>> Android+iOS dwarfed whatever marketshare Windows has.
>> 
>> If userbase is what matters to you, then OpenRC+eudev won. It is the
>> logical choice for those concerned about userbase because that is what
>> the Linux ecosystem will be using going forward.
>> 
>> I do not think userbase should be the sole means by which we make
>> decisions, but those that think otherwise should now join the
>> eudev+OpenRC camp. It has the bigger userbase share going forward.
>> 
>> To put it another way, the war is over. Welcome abroad. :)
> 
> Oh, the new thing every cool kid users these days. I have only one question 
> in return: for how long?
> 
> Today Alpine uses eudev. But people change, distributions change. One year 
> from now, it may be using systemd.
> 
> Today docker uses Alpine. Tomorrow it may use something else. Or even require 
> systemd by design.

The Alpine Linux developers use eudev where mdev is a pain to use and prefer 
mdev to it. They also prefer busybox init to just about everything else. The 
only way Alpine Linux would use systemd is if it were merged into busybox, but 
the busybox developers dropped systemd support last year because the systemd 
developers were not great at collaborating. The "busybox is a joke" remarks 
that I got from them before that happen means that they have been actively 
sabotaging that relationship for a long time.

I think it is safe to assume that Redhat will drop Linux for Windows before 
Alpine Linux uses systemd.

> Today docker is the cool thing. One year from now, nobody may remember about 
> it. Didn't things like this happen before?

The iPhone did the same thing to Windows. Then Android came along. Neither 
contribute to Windows hemomgeny, which is dead. The same is true for systemd. 
Coincidentally, neither Android nor iOS use systemd.

> Now, let's extend this to a perspective of few years. What is more likely to 
> be extinct: userbase of eudev or systemd?

systemd will always have its niche. The same is true for eudev. The latter will 
always exist as long the the former exists though as there are too many people 
who have been alienated by systemd development for them to accept a system 
running it.

That is unlikely that will change unless developers who the systemd developers 
alienate were to all die at once.

>>>> 
>>>> it needs to be in the new stage4s to make a bootable system.  imo a
>>>> stage4 should be bootable modulo a kernel.
>>> 
>>> Sure, a stage4 based on systemd makes a lot of sense.  I don't really
>>> see the point in leaving a kernel out though - I'd even stick a
>>> precompiled one in /boot on top of having the sources installed.  Why
>>> not make a stage4 install something that takes all of 5 minutes?
>>> 
>>> I think that offering an eudev-based distro as a default just doesn't
>>> make sense in 2016.  I just think the better road to take is to start
>>> treating virtual/udev as something that gets installed post-stage3.
>>> We can't even get people to agree on vi vs emacs as a default.
>> 
>> We can leave virtual/udev out of stage3, but that doesn't diminish the
>> need to select sensible default for the virtual/udev provider.
> 
> 
> -- 
> Best regards,
> Michał Górny (by phone)
> 

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

> On Feb 17, 2016, at 7:25 AM, Rich Freeman <ri...@gentoo.org> wrote:
> 
>> On Tue, Feb 16, 2016 at 11:00 PM, Richard Yao <r...@gentoo.org> wrote:
>> 
>> If userbase is what matters to you, then OpenRC+eudev won. It is the
>> logical choice for those concerned about userbase because that is what
>> the Linux ecosystem will be using going forward.
> 
> Uh, if we cared solely about userbase we'd be using upstart.  I'm
> pretty sure that there are more chromebooks out there than Docker
> containers running Alpine linux.
> 
> Do people actually use docker images created by docker?

They do. Docker is popular because it enables developers to start microservices 
from prebuilt images. Those images by default are Alpine Linux, which means 
anything using udev in them is using eudev.

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

> On Feb 17, 2016, at 7:43 AM, Michał Górny  wrote:
> 
> On Tue, 16 Feb 2016 23:41:33 +0100
> Chí-Thanh Christopher Nguyễn  wrote:
> 
>> Alexis Ballier schrieb:
> If it's just that, it's not limited to udev, but anything using
> kdbus/bus1, and would mean openrc/${favorite init system} will have
> to do the same thing anyway. But again, almost 2 years is extremely
> old considering all the flux that has been around kbus.  
 
 OpenRC itself can for now just ignore kdbus, bus1, or whatever kernel
 IPC system comes next.  
>>> 
>>> Well, as Lennart wrote it, kbus would have needed some initialisation.
>>> Just like we have a dbus init script, openrc would have a kdbus one.
>>> 
 But if upstream udev makes use of the systemd
 userspace interface to the kernel IPC system, then OpenRC would have
 to implement the same interface in order to have working udev.  
>>> 
>>> As I understand it, a kernel IPC doesn't need systemd to work. udev
>>> might use wrappers from libsystemd, or libbus1, just like we have
>>> programs using libv4l or libbluetooth currently.  
>> 
>> In a follow-up, upstream wrote about how you should only run udev together 
>> with systemd, and if you don't want to do that (spelling as in original):
>> 
>> "we will not support the udev-on-netlink case anymore. I see three options:
>> a) fork things, b) live with systemd, c) if hate systemd that much, but
>> love udev so much, then implement an alternative userspace for kdbus to
>> do initialiuzation/policy/activation."
>> https://lists.freedesktop.org/archives/systemd-devel/2014-May/019664.html
>> 
>> So it seems a bit more than only initialization is needed.
> 
> You're missing the third option which is a sane option, and jump
> straight to pitchforks.
> 
> As I see it, *if* this becomes a necessity, we're quite like are going
> to provide KDBUS parts of systemd the way we provide udev parts right
> now. After all, libsystemd-bus will be useful to more applications.
> 
> Of course, someone may want to fork that into libebus just for the sake
> of renaming.
> 
> And after all, as it has already been noted, there are people
> interested in maintaining non-systemd userspace for KDBUS. Which is
> kinda the obvious choice, unlike forking something.

kdbus is dead. It is fatally flawed and Greg is no longer trying to get it 
merged as he is not updating his branch for newer kernel versions. If I recall 
correctly, kdbus was also removed from Fedora and has no distribution backing 
it anymore.

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

On 02/08/2016 10:09 PM, Rich Freeman wrote:
> On Mon, Feb 8, 2016 at 7:58 PM, Anthony G. Basile
 wrote:
>>
>> what does in-house tool mean?  i'm a gentoo developer but i also work
>> on an upstream project (eudev) that 14 distros use.
>>
>> some of the criticism given here are my concerns as well and i've
>> spoken with the various distros --- slack, parted magic, puppy.  they
>> get what's going on and they still see eudev is the best way forward
>> for now.  it may not be in the future, but neither will a udev
>> extracted from a compiled full systemd codebase.
>
> How many of those 14 distros have more than 14 users?
>
> Look, I get it, some people don't like systemd.  That's fine.
> However, you have to realize at this point that a non-systemd
> configuration is anything but mainstream.  There will always be a
> "poppyseed linux" whose purpose in life seems to be to preserve linux
> without sysfs or some other obscure practice.  I just think that
> Gentoo should offer the choice to do those things, but have a more
> mainstream set of defaults.

The new mainstream is docker. Docker recently switched to Alpine Linux,
which uses OpenRC+eudev:

https://www.brianchristner.io/docker-is-moving-to-alpine-linux/

That dwarfs whatever marketshare systemd has in the same way that
Android+iOS dwarfed whatever marketshare Windows has.

If userbase is what matters to you, then OpenRC+eudev won. It is the
logical choice for those concerned about userbase because that is what
the Linux ecosystem will be using going forward.

I do not think userbase should be the sole means by which we make
decisions, but those that think otherwise should now join the
eudev+OpenRC camp. It has the bigger userbase share going forward.

To put it another way, the war is over. Welcome abroad. :)

>>
>> it needs to be in the new stage4s to make a bootable system.  imo a
>> stage4 should be bootable modulo a kernel.
>>
>
> Sure, a stage4 based on systemd makes a lot of sense.  I don't really
> see the point in leaving a kernel out though - I'd even stick a
> precompiled one in /boot on top of having the sources installed.  Why
> not make a stage4 install something that takes all of 5 minutes?
>
> I think that offering an eudev-based distro as a default just doesn't
> make sense in 2016.  I just think the better road to take is to start
> treating virtual/udev as something that gets installed post-stage3.
> We can't even get people to agree on vi vs emacs as a default.

We can leave virtual/udev out of stage3, but that doesn't diminish the
need to select sensible default for the virtual/udev provider.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

On 02/08/2016 07:46 AM, Michał Górny wrote:
> On Mon, 8 Feb 2016 10:08:22 +0100
> Patrick Lauer  wrote:
> 
>> Ohey,
>>
>> I've opened a bug at:
>> https://bugs.gentoo.org/show_bug.cgi?id=573922
>>
>> The idea here is to change the order of the providers of virtual/udev.
>> For existing installs this has zero impact.
>> For stage3 this would mean that eudev is pulled in instead of udev.
>>
>> The rationale behind this is:
>>
>> * eudev is an in-house fork, and there's more than a dozen distros
>> already using it by default that are not us. Which is a little bit weird ...
> 
> That's not an argument. I can also fork random system components. Would
> you consider that a reason to replace the defaults with our 'in-house'
> forks?
> 
>> * Both udev and eudev have pretty much feature parity, so there won't be
>> any user-visible changes
>>
>> * udev upstream strongly discourages standalone udev (without systemd)
>> since at least 2012
>>
>> (see for example:
>> https://lists.freedesktop.org/archives/systemd-devel/2012-June/005516.html
>> https://lkml.org/lkml/2012/10/3/618
>> )
>>
>> So it'd be (1) following upstreams recommendations and (2) dogfooding
>> our own tools. I don't see any downsides to this :)
> 
> I'm strongly against this, because:
> 
> 1. It is a conflict-induced fork. As such, it will never be merged
> upstream and it will never be supported upstream. In fact, it is
> continually forces to follow upstream changes and adapt to them. eudev
> is more likely to break because of the Gentoo developer(s) working hard
> to merge upstream changes to their incompatible code.

That was the entire point of the project. Upstream rejected any attempts
to do things that we actually needed and broke things claiming the
distributions were responsible for handling the breakage, so eudev was
started on the basis that we needed a project that would ensure that
changes in udev occur in a way that makes sense.

> 2. Many of Gentoo users are programmers who appreciate the 'vanilla'
> API experience Gentoo often provides. Switching the defaults to a fork
> that is known to intentionally diverge from upstream goes against that
> principle. Programs written against eudev may not work correctly with
> upstream udev.

If upstream udev were stable, that would be one thing, but it
intentionally diverges from itself continuously. The only experience
that could be reliably provided with upstream udev is one of continual
breakage.

> 3. eudev has fallen behind systemd/udev more than once in the past,
> and caused visible breakage to users this way.

When?

Can we also consider all of the times udev broke the boot process
because upstream just didn't care about doing changes in a sane way and
the people interested in providing the upstream experience delivered on
that goal?

> 4. eudev is underdocumented, and the maintainer admits that 'he sucks
> at documenting'. In fact, did anyone even bother to note how far eudev
> diverges from upstream udev to this point?

The FreeBSD developers were complaining about how poorly documented udev
was well before eudev existed. This is not a regression unless systemd's
innovations in replacing documented things with undocumented things made
them worse.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Changing order of default virtual/udev provider

[Richard Yao]

On 02/08/2016 04:08 AM, Patrick Lauer wrote:
> Ohey,
> 
> I've opened a bug at:
> https://bugs.gentoo.org/show_bug.cgi?id=573922
> 
> The idea here is to change the order of the providers of virtual/udev.
> For existing installs this has zero impact.
> For stage3 this would mean that eudev is pulled in instead of udev.
> 
> The rationale behind this is:
> 
> * eudev is an in-house fork, and there's more than a dozen distros
> already using it by default that are not us. Which is a little bit weird ...
> 
> * Both udev and eudev have pretty much feature parity, so there won't be
> any user-visible changes
> 
> * udev upstream strongly discourages standalone udev (without systemd)
> since at least 2012
> 
> (see for example:
> https://lists.freedesktop.org/archives/systemd-devel/2012-June/005516.html
> https://lkml.org/lkml/2012/10/3/618
> )
> 
> So it'd be (1) following upstreams recommendations and (2) dogfooding
> our own tools. I don't see any downsides to this :)
> 

+1



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Change in Gentoo ZFS packaging policy

[Richard Yao]

Fix fail path in zfs_znode_alloc
  Fix snapshot automount behavior when concurrent or fail
  Linux 4.4 compat: make_request_fn returns blk_qc_t
  Linux 4.4 compat: xattr operations takes xattr_handler
  Fix zdb calling behavior in ztest
  Fix zdb_dump_block on little endian systems
  Use spa as key besides objsetid for snapentry
  Fix snapshot automount race cause EREMOTE
  Fix uio_prefaultpages for 0 length iovec
  Fix empty xattr dir causing lockup
  Make xattr dir truncate and remove in one tx

DHE (1):
  libzfs: handle EDOM errors

Jason Zaman (1):
  sysmacros: Make P2ROUNDUP not trigger int overflow

Kamil Domański (1):
  Skip GPL-only symbols test when cross-compiling

Ned Bass (1):
  Prevent SA length overflow

Olaf Faaland (1):
  Remove "index" column from dbufstat.py

Richard Yao (5):
  ClusterHQ Proposed API extensions
  Only trigger SET_ERROR tracepoint event on error
  Unconditionally build zdb and ztest with -DDEBUG
  Revert "Switch ztest mmap(2) ASSERTs to VERIFYs"
  Refresh ClusterHQ Proposed API extensions against ZoL PR

Turbo Fredriksson (1):
  Unmount is part of the shutdown process, not the boot process.

tuxoko (3):
  Fix null pointer in arc_kmem_reap_now on 32-bit
  Fix zfs_dirty_data_max overflow on 32-bit
  Prevent rm modules.* when make install

The SPL shortlog:

Brian Behlendorf (1):
  Fix --enable-linux-builtin

Chunwei Chen (1):
  Don't call kmem_cache_shrink from shrinker

Dimitri John Ledkov (1):
  spl-kmem-cache: include linux/prefetch.h for prefetchw()

Jason Zaman (1):
  sysmacros: Make P2ROUNDUP not trigger int overflow

Richard Yao (2):
  Fix race between getf() and areleasef()
  Make taskq_member() use ->journal_info

tuxoko (2):
  Fix taskq dynamic spawning
  Prevent rm modules.* when make install


0xBEE84C64.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

[gentoo-dev] New license: CROSSOVER-3

[Richard Yao]

Earlier this year, I spoke with Codeweavers' CEO, Jeremy White, about
changing their license terms so that we could remove RESTRICT=fetch from
app-emulation/crossover-bin. Robin Johnson and I also discussed it then.

The main issue was the requirement that users delete the software after
the trial period and Jeremy agreed to make a change to the crossover
license when Crossover 15.0.0 was released. This just happened and the
change occurred as discussed.

I have attached the new license and its diff. There are only two changes:

1. They now claim to bundle copies of libxml2 and libxslt under
BSD-style licenses.
2. The section on deleting the software has been deleted as per our
discussion.

Does anyone have any objections to committing the new license?
--- /usr/portage/licenses/CROSSOVER-2   2015-08-09 16:34:52.0 -0400
+++ CROSSOVER-3 2015-12-08 12:15:42.749265174 -0500
@@ -52,6 +52,7 @@
   www.icculus.org.
 
   We also use or include static copies of the following projects:
+libxml2, libxslt- BSD-style
 xml-dom, xml-namespacesupport,  - Artistic or GPL license
 xml-regexp, xml-sax
 
@@ -119,10 +120,7 @@
 
   If the Software was given to you for purposes of evaluation, then this
   License will terminate at the end of the specified evaluation period,
-  typically 30 days.
-
-  Upon termination you must destroy the Software, related documentation and
-  all copies thereof.
+  typically 15 days.
 
5. Export Law Assurance. You agree and certify that neither the Software nor
   any other technical data received from VENDOR, nor the direct product
CrossOver Linux License Grant

YOU REALLY WANT TO READ THIS, ESPECIALLY THE PART ABOUT
THE MANDATORY CAR WASH FOR CODEWEAVERS EMPLOYEES...

If you don't like this license grant:

   a. Let us know, we'd appreciate the feedback.

   b. Stop right now, and ask for a refund. We'll cheerfully do so.


The main thing we want you to know:
 This is a license for one user. The license is not necessarily for a
specific user, or a specific computer, but it is for one person at a
time. If you need to support more than one person, please contact us
for volume pricing and site licensing.  We do offer educational
discounts.

< Start of Formal License Grant >---

   1. License. The software accompanying this License (hereinafter "Software"),
  regardless of the media on which it is distributed, are licensed to you
  by CodeWeavers ("VENDOR"). You own the medium on which the Software is
  recorded, but VENDOR and VENDOR's Licensors (referred to collectively as
  "VENDOR") retain title to the Software and related documentation. You
  may:

 a. run the Software on any computer, so long as no more than one
person per license is ever using the Software at any one time.

 b. transfer all your license rights in the Software, the backup copy
of the Software, the related documentation and a copy of this
License to another party, provided the other party reads and agrees
to accept the terms and conditions of this License.


   2. Free Software. The Software contained in this product includes some
  components of Free Software, including software from the Wine Project,
  and the MojoSetup setup software.

  The Wine project is licensed under terms of the GNU Lesser General Public
  License, which is included below as Appendix A. The best source for the
  Wine source code is the main Wine web page at http://www.winehq.org.

  Japanese fonts are included under the Wada Laboratory public domain
  license found at
  http://sourceforge.jp/projects/ume-font/wiki/UmeFontLicence

  MojoSetup and its dependent projects are all licensed under BSD
  style licenses.  The best place for that code is also
  www.icculus.org.

  We also use or include static copies of the following projects:
libxml2, libxslt- BSD-style
xml-dom, xml-namespacesupport,  - Artistic or GPL license
xml-regexp, xml-sax

  We also use the htmltextview.py library by Gustavor Carneiro, which
  is licensed under the LGPL.  The source code was available to us at:
http://people.gnome.org/~gjc/htmltextview.py

  Portions of this software are copyright © 2009. The FreeType
  Project (www.freetype.org).  All rights reserved.

  In each case, we use them unmodified and strongly recommend that
  anyone wishing source code for these projects find and visit the
  respective project home page.

  We are deeply grateful to the authors of all of these software projects
  for allowing us to use their software.

  We include source code with each CD purchase of CrossOver. Current
  source code for Free Software contained within CrossOver products is 

Re: [gentoo-dev] Introduce ppc64le architecture into gentoo ! please share your comments

[Richard Yao]

On 10/21/2015 04:55 AM, Kevin Zhao wrote:
> Hi Guys， We have finish compiling stage3 for ppc64  (little-endian).Here is
> the link:
> 
> https://drive.google.com/file/d/0B2k84p6709AyTFlwLUF1WjlxUk0/view?usp=sharing

Awesome!

>  Now we are going to build LiveCD using stage3.  Could you help to give
> some demo or a guide for building LiveCD? That will help we a lot to make
> effort.
> Thanks ~

I suggest asking in #gentoo-releng on freenode or the Gentoo release
engineering mailing ilst.
> 
> 
> 2015-09-27 3:16 GMT+08:00 Anthony G. Basile :
> 
>> On 9/24/15 8:23 AM, Leno Hou wrote:
>>
>>>
>>>
>>>   Again, please apply POWER8 Systems and join our work  :-)
>>>   [1]: https://www.runabove.com/instances/ibm-power8.xml
>>>   [2]: https://ptopenlab.com/cloudlabconsole/#/
>>>   [3]: http://osuosl.org/services/powerdev/request_hosting
>>>
>>>
>>> -Leno Hou


>> I have applied to osuosl for an ibm power8 ppc64le node.  I've put myself
>> down as the point-of-contact.  I asked for openstack gui access and will
>> try to build a system using Leno's stage3.  I'm not sure what the env looks
>> like right now, but i assume i'll have to boot off a cd image.  I'll use
>> debians.  I guess I could be macho and try to build everything from
>> scratch, cross compile a kernel and all, but   time.
>>
>> Has anyone tried qemu simulating ppc64le on amd64?  Lu I thought you said
>> you tried on x86 and it didn't work.
>>
>>
>>
>> --
>> Anthony G. Basile, Ph.D.
>> Gentoo Linux Developer [Hardened]
>> E-Mail: bluen...@gentoo.org
>> GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
>> GnuPG ID  : F52D4BBA
>>
>>
> 




signature.asc
Description: OpenPGP digital signature

[gentoo-dev] What is the status of phone/tablet support?

[Richard Yao]

I asked in #gentoo-embedded on freenode, but I would like to pose this question
to a wider auidence through the list.

What is the status of phone/tablet support? In specific, I am curious about:

* Modern hardware options (especially those without a hardware keyboard)
* Status of F/OSS drivers
* Xorg touch support (multitouch, virtual keyboards, etcetera)
* Desktop environments designed for multitouch in portage
* Graphical applications (e.g. dialer, calendar, web browser, email client)
* Reliable filesystem options for flash storage in small memory environments.

I know that the KDE project has Plasma Mobile, which gives us both a desktop
environment and many key graphical applications. However, I am not sure if much
has been done to support it in Gentoo's packaging and I would like to hear about
this:

http://www.youtube.com/watch?v=auuQA0Q8qpM

I realize that it is difficult to convey the sort of information that one wants
on an open ended topic such as this, so I will try to lead by example by talking
about filesystem options. First off, Linaro has some good summary:

https://wiki.linaro.org/Flash%20memory

Their summary consists entirely of log-based filesystems and CoW filesystems. It
omits in-place filesystems such as ext4 and XFS. In-place filesystems really
should not be appropriate for a phone or tablet because they are not designed to
gracefully recover from unclean unmounts caused by kernel panics or power loss
(e.g. dead battery, battery removal), although Android uses ext4 anyway.

A notable exception from Linaro's list that I happen to maintain in the tree is
the ZFSOnLinux driver. ZFS has many attributes that are desireable for phones,
such as easy backup through incremental send/recv (easy on bandwidth and battery
life), the ability to detect corrupt data (flash is not perfect), LZ4
compression (better throughput/latencies and less flash is need to store the
amount data) and ARC for a better cache hit rate when your working set exceeds
the amount of memory you have for cache (always true on phones).

ZFSOnLinux is currently behind other CoW filesystems in support for
phones/tablets. Upstream is gradually improving this and it should become a
competitive option in the future. The main areas that pose problems are:

1. Efficient use of minimal memory.

1a. Long held objects in SLABs cripple reclaim efficiency by preventing mostly
empty slabs from being reclaimed. On Solaris, some of the caches implement a
move callback that allows for compaction, but Linux's SLAB implementation does
not provide the option, ZoL's SLAB implementation does not implement it and it
is not clear if it is possible to defragment the things on Linux that use it
without modifications to the Linux kernel. Thankfully, nearly all of the memory
used by SLABs is in the ZIO buffers, which will be replaced with page-based
scatter-gather lists in the next release. That will go a long way to mitigating
this.

1b. mmap() double caches between the ARC and the page cache. This has been done
in OSv's ZFS driver and can be done in the ZFSOnLinux driver. It is currently
blocked on 1a as it does not make sense to implement sooner, but it will be
definitely be done.

2. 32-bit support (not an issue on 64-bit ARMv8 hardware, but ARMv8 is rare)

2a. Linux's kernel virtual memory allocator is crippled because the kernel
enters an infinite loop when it runs out of address space set aside for kernel
virtual memory. This is not a problem on 64-bit system swhere the amount set
aside exceeds the amount any single image system in existence currently has, but
it is a major problem on 32-bit. There is the further problem of a global
spinlock being used and certain accoutning functions iterating a linked list of
all virtual memory allocaitons, but this secondary has been mitigated mostly
through SLAB allocation. The main use of kernel virtual memory is to
avoid external fragmentation issues that come from allocting large buffers,
especially the SLABs that are used to minimize the number of kernel virtual
memory allocations that are done. The next release will implement scatter-gather
lists that should bypass this on the ZIO bufers while the other consumers either
have already been moved to kernel virtual memory or should be moved.

2b. There is no upstream test coverage on 32-bit and there are open issues
involving 32-bit runtime failures. The project has gradually fixed 32-bit
issues, but more work needs to be done here to make 32-bit as reliable as
64-bit. This will likely come after the solution to 2a makes 32-bit feasible
enough to subject it to regular testing.

3. Tweaks for flash storage.

3a. We will gain discard support from the Illumos community in the next release,
but the behavior of the hardware when discard is translated into the native
command needs to be studied. On SATA, discard is mapped to ATA TRIM. The design
of ATA TRIM is broken before SATA 3.1 where TRIM is an unqueued command that
introduces arbitrary lags. Consequently, 

Re: [gentoo-dev] Add bc back to the stage3

[Richard Yao]

On 09/27/2014 07:39 PM, Anthony G. Basile wrote:
 On 09/27/14 18:46, Rich Freeman wrote:
 On Sat, Sep 27, 2014 at 5:52 PM, Tom Wijsman tom...@gentoo.org wrote:
 What is really needed here is a vote by the Council on whether to add bc
 back to the stage3. If the people do insist, another vote regarding
 adding or changing an editor to stage3 could be done as well.

 The call for agenda goes out on Tuesday, so if somebody wants a vote
 please put it up.  Don't let mgorny be the only one with agenda items.
 :)

 -- 
 Rich

 He isn't ... remember GLEP 64 :)
 
 And now for something completely different ... drum roll ... Really!  We
 have to have a council vote on whether bc goes into stage3?  If this
 does go to the council, then I want a pre-vote vote: should we bounce
 the decision back to the releng team?  We should not micro manage to
 this level.
 

May I suggest an alternative? We could implement sys-virtual/posix and
make it depend on all packages that are not necessary for @system, but
are necessary for proper POSIX compliance. Then we can tell users who
need/want an environment containing all tools specified by POSIX, such
as those not using sys-kernel/*-sources, to `emerge virtual/posix`.

That said, the larger matter of standards conformance that needs to be
considered. Illumos' Garrett D'amore has been working on standards
conformance tests for libc:

https://bitbucket.org/gdamore/illumos-gate/src/8815a50c9cc3f6f213931e12f72c252504363a82/usr/src/test/libc-tests/?at=core

Garrett told me yesterday that the changes necessary to run them on
Linux should be very small and are likely restricted to a few dozen
lines in 1 file. I want to try running them to catch POSIX conformance
issues in our base system. That will likely come later this year, as I
recently became aware of a SUS conformance issue in ZFS' implementation
of mmap() where PROT_WRITE + MAP_PRIVATE on a readonly file fails.
Fixing that will take priority over reviewing the standards conformance
of libc (others can review libc before i do if they wish).

I imagine that the tests will catch issues in our present conformance
when they are run. Once we have the results, we will need to decide how
proactive we intend to be about fixing them. We will definitely want to
work with upstream libcs to get issues that tests identified fixed.
However, there would be the question of whether we wish to fix them
immediately or wait for the patches fixing them to be upstreamed. If the
matter of adding bc to the base system for POSIX conformance goes to the
Council, it might be worth thinking about how far we wish to go for
standards conformance when further issues are identified.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: new set of profiles/default/linux/uclibc

[Richard Yao]

On Thu 14 Aug 2014 12:04:10 PM EDT, Anthony G. Basile wrote:
 Hi everyone,

 I'm going to add a new set of profiles under default linux.  These
 will mirror what's already under hardened and will have the structure:

 default/linux/uclibc/{amd64,arm,mips,ppc,x86}

 Currently I maintain both hardened and non-hardened uclibc stages for
 all those arches.  I've been housing the vanilla profiles under
 hardened by just turning off USE=hardened, but that's hacky.  Rather,
 we should have default/linux/uclibc and have hardened/linux/uclibc
 inherit from them like we do with default/linux in general.


This sounds good to me.



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: [gentoo-project] Returning dev: Thomas Alan Gall (tgall)

[Richard Yao]

Welcome back! :)

On 07/14/2014 04:26 AM, Justin (jlec) wrote:
 Hi everyone,
 
 we have an returning oldtimer here, Thomas Gall aka. tgall. His original
 bug has been opened in 2003, so he knows gentoo from the early days.
 
 He is joining the arm team now and will stabilize mostly for arm64,
 where he already started to work with a rate of  10 pkgs/day. Happy
 days for arm users!
 
 Some biographic notes from himself:
 
 Reenactor of War of 1812, Napoleonic French and US Civil War eras. Long
 time software engineer currently working for Linaro as the acting
 director of the Linaro Mobile Group. Former and now returned Gentoo dev
 interested now days in arm and arm64. Former Mayor of Mantorville. Dad,
 husband and general jack of all trades.
 
 
 Please join me in welcoming him back!
 
 Justin
 




signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: GSoC proposal: cp --reflink support for zfs.

[Richard Yao]

A key feature of reflinks is that they operate on any data in a mountpoint, but 
what you described only applies to data with a deduplication table entry. In 
such cases, it do not see what it accomplishes over simply using data 
deduplication. In specific, there is no efficiency advantage. It is not clear 
to me that trying to cut corners to obtain one is possible without incurring 
consistency issues with the deduplication table falling out of sync. In the 
case that there is no deduplication table entry, you would need to rewrite the 
file. reflinks are intended to be as done as quickly as hard links, so this 
would seem to negate the benefit.

Matthew Ahrens and I have discussed reflinks in the past and we both agree that 
they would be non-trivial to implement. That does not mean it cannot be done, 
but I do not think this particular approach would succeed. However, I encourage 
you to keep thinking about such things. If you think of a way of doing this 
that seems workable, it would likely be something that could be made into a 
GSoC project.

With that said, if you want to do a ZoL-related project for Gentoo, I have some 
other things that I could suggest that I believe are workable. Such ideas are 
things that I was asked to prepare on extremely short notice and I have not yet 
had time to publish them. Let me know if you would be interested.

On Mar 12, 2014, at 3:15 AM, Yuxuan Shui yshu...@gmail.com wrote:

 Hi,
 
 I would like to implement cp --reflink support for ZFSOnLinux as my GSoC 
 project.
 
 cp --reflink is used to create a COW copy of a file, so the file will not 
 take any disk space if it's not modified. This feature is very useful for 
 cases like storing a lot of almost identical virtual machine images. Also 
 this is a frequently requested feature for ZoL. [1][2][3]
 
 Currently only btrfs support this feature, so my goal it to bring it to ZoL 
 as well.
 
 I think the only way to do it (without changing too many parts of ZoL) is to 
 use the deduplication feature of zfs. A COW copy could be done by create a 
 new entry in ddt for the old file, and create a new file which points to the 
 ddt entry.
 
 Please let me know if this proposal makes sense, and if that's the right way 
 to do it.
 
 Thanks.
 
 [1]: 
 https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/mvGB7QEpt3w
 [2]: https://github.com/zfsonlinux/zfs/issues/405
 [3]: https://github.com/zfsonlinux/zfs/issues/1063
 -- 
 
 Regards
 Yuxuan Shui

Re: [gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[Richard Yao]

On 03/12/2014 08:45 AM, Alex Xu wrote:
 On 12/03/14 03:15 AM, Yuxuan Shui wrote:
 Hi,

 I would like to implement cp --reflink support for ZFSOnLinux as my GSoC
 project.

 cp --reflink is used to create a COW copy of a file, so the file will not
 take any disk space if it's not modified. This feature is very useful for
 cases like storing a lot of almost identical virtual machine images. Also
 this is a frequently requested feature for ZoL. [1][2][3]

 Currently only btrfs support this feature, so my goal it to bring it to ZoL
 as well.

 I think the only way to do it (without changing too many parts of ZoL) is
 to use the deduplication feature of zfs. A COW copy could be done by create
 a new entry in ddt for the old file, and create a new file which points to
 the ddt entry.

 Please let me know if this proposal makes sense, and if that's the right
 way to do it.

 Thanks.

 [1]:
 https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/mvGB7QEpt3w
 [2]: https://github.com/zfsonlinux/zfs/issues/405
 [3]: https://github.com/zfsonlinux/zfs/issues/1063

 
 While I can't comment too much on the technical aspects, they seem to be
 relatively sound.
 
 However, there are some issues with the, er... other aspects, for lack
 of better terminology.
 
 1. This is possibly out of scope as a Gentoo project, since ZOL is not
 really part of Gentoo. If it's not, then you're out of luck, because ZOL
 is not an accepted organization.

Things that provide us with improvements over what we have are
definitely worth consideration as GSoC projects. However, what is
accepted ultimately depends on not only feedback from a potential
mentor, but also a vote of Gentoo developers.

 2. This is likely too small to be a GSoC project. Perhaps see [0] for a
 list of example ideas, if only so you can get a grasp on the size of a
 good project.
 
 It does sound like a good idea though, and even if you can't do it as
 part of GSoC, you should pursue it anyways.

Leaning on my understanding of ZFS internals, I can say that this is
large enough. However, I do not think it will accomplish the desired
result if implemented in the manner suggested.



signature.asc
Description: OpenPGP digital signature