Re: [RFC] prctl: (Please do not!) Deprecate non PR_SET_MM_MAP operations

2018-05-20 Thread Jonathan de Boyne Pollard
In case you weren't aware, programs can make use of a facility on Linux 
to make their ps listings contain appropriate command-line arguments and 
environment strings. There's also a similar kernel API on FreeBSD.  This 
is used by the built-ins in nosh to good effect, even more so in the 
forthcoming version 1.38, allowing system adminstrators to debug things 
more easily from ps listings by showing them the actually effective 
states of the chained argument and environment vectors rather than what 
they were at the beginning of a chain of built-in commands.


* https://unix.stackexchange.com/a/438007/5132

* https://unix.stackexchange.com/a/432681/5132

The subject of deprecating and removing the API that permits this came 
up recently on the Linux Kernel Mailing List.  I raised my concerns, and 
the Linux Kernel developers have been laudably responsive on the 
subject.  As I said to them, I would welcome a better API, that allows 
setting these things atomically, start and length together as a pair.


* https://lkml.org/lkml/2018/4/3/739

* https://lkml.org/lkml/2018/5/6/127

* https://lkml.org/lkml/2018/5/7/78

* https://lkml.org/lkml/2018/5/7/79

* https://lkml.org/lkml/2018/5/7/80

* https://lkml.org/lkml/2018/5/7/81


s6-rc live state and database format upgrades

2018-05-20 Thread Guillermo
Hello.

I was recently reviewing the instructions for the s6-rc dabatase
upgrade procedure, and they made me wonder what would happen if:

* s6-rc-format-upgrade was called with a compiled database that is not
the exact 0.4.0.x equivalent of the one currently associated with the
live state directory (i.e. other that a database created with
s6-rc-compile from version 0.4.0.x and the exact same service
definitions).

* s6-rc-update from version 0.4.0.x was called with a live state
directory currently associated with a <=0.3.0.0 database.

Is any of these things capable of trashing s6-rc's live state? I think
that the documentation is clear about the upgrade procedure, but that
these could be likely ways of accidentally screwing it up. Something
like the latter was actually the database upgrade procedure for
previous backwards-incompatible s6-rc upgrades, right? Although I
don't know if any of them involved a database format change.

Also, I'm lost about the role of version 0.3.0.1:

* https://www.mail-archive.com/skaware@list.skarnet.org/msg01096.html
* 
https://git.skarnet.org/cgi-bin/cgit.cgi/s6-rc/tree/doc/upgrade.html?h=real_0.3.0.1

Development of 0.3.0.1 was done in the skarnet.org Git repository in a
'real_0.3.0.1' branch. Development of 0.4.0.0 was done in the master
branch, and the current upgrade notes don't even mention 0.3.0.1:

* http://www.skarnet.org/software/s6-rc/upgrade.html

So, in the end, does upgrading from 0.3.0.0 to 0.4.0.x require an
intermediate 0.3.0.1 s6-rc-compile + s6-rc-update step, and then
0.4.0.x s6-rc-compile + s6-rc-format-upgrade, or not?

(I didn't have running services that I had to preserve, so personally
I didn't need any special care and just upgraded from whatever I had
at the time to s6-rc-0.4.0.0, but s6-rc-format-upgrade is now there
and one might need to explain or remind correct usage :) )

Thanks,
G.


Re: [RFC] prctl: (Please do not!) Deprecate non PR_SET_MM_MAP operations

2018-05-20 Thread Laurent Bercot
The subject of deprecating and removing the API that permits this came 
up recently on the Linux Kernel Mailing List.  I raised my concerns, 
and the Linux Kernel developers have been laudably responsive on the 
subject.  As I said to them, I would welcome a better API, that allows 
setting these things atomically, start and length together as a pair.


 Thanks for doing the necessary and often ungrateful work of keeping
kernel developers in check, Jonathan. ;)

--
 Laurent



Re: s6-rc live state and database format upgrades

2018-05-20 Thread Laurent Bercot

* s6-rc-format-upgrade was called with a compiled database that is not
the exact 0.4.0.x equivalent of the one currently associated with the
live state directory (i.e. other that a database created with
s6-rc-compile from version 0.4.0.x and the exact same service
definitions).


 It would probably mess up your live state and your dependencies, badly.
You would need to manually reboot to fix it.



* s6-rc-update from version 0.4.0.x was called with a live state
directory currently associated with a <=0.3.0.0 database.


 The update would not work: 0.4.0.x s6-rc, invoked from s6-rc-update,
would choke on your old database and exit with an error message.



Is any of these things capable of trashing s6-rc's live state? I think
that the documentation is clear about the upgrade procedure, but that
these could be likely ways of accidentally screwing it up. Something
like the latter was actually the database upgrade procedure for
previous backwards-incompatible s6-rc upgrades, right? Although I
don't know if any of them involved a database format change.

Also, I'm lost about the role of version 0.3.0.1:


 What happened, in summary:

 * 0.4.0.x is a database format change over 0.3.0.x. The 0.3 format
implements pipelines, but the 0.4 format implements funnels instead.

 * The simplest way of upgrading the format is just... to put the
new database in place of the old one, without any conversion whatsoever.
That is what the 0.4.0.x s6-rc-format-upgrade does. It blindly replaces
the $livedir/compiled symlink with a symlink to the new database. It
does zero magic: the magic happens earlier. (But you should not rely on
this, because s6-rc-format-upgrade MAY do magic in the future.)

 * Simple replacement can only work iff the new db and the old db
represent the same set of services AND those services have the same
numbers. IOW: the old db contents and the new db contents are exactly
the same, only the format changes.

 * Up to 0.3.0.0, s6-rc-compile does not guarantee the numbers for the
services - it assigns the numbers as it finds the source definition
directories with readdir(). So it's impossible to make sure that a
0.4.0.x db can be a drop-in replacement.

 * That is why I released 0.3.0.1, where s6-rc-compile actually uses
deterministic numbering of its services: the same source db will always
produce the same compiled db. (It's just a question of adding a
qsort() when you have all the service names in a source directory. :))
This is not official or part of the public API: users aren't supposed to
know or use service numbers, or rely on deterministic numbering.

 * 0.4.0.x uses the same deterministic numbering. So, the same source
db compiled with 0.3.0.1 and 0.4.0.x will produce the same compiled
databases with only the format changing, and those dbs are suitable
to be exchanged with s6-rc-format-upgrade.

 * The 0.3.0.1 stepping stone was only necessary because prior to it,
service numbering was not deterministic. Now that it is deterministic,
the intermediary step will not be necessary for future format upgrades.
IOW: you will be able to switch directly from 0.4.0.x to 0.5.0.0. Or
from 0.3.0.1 to 0.5.0.0.



Development of 0.3.0.1 was done in the skarnet.org Git repository in a
'real_0.3.0.1' branch. Development of 0.4.0.0 was done in the master
branch


 Yes, because I first worked on 0.4.0.0, in the master branch, and
then realized that a format change would break people's live states,
and a clean upgrade path was needed. So I wrote s6-rc-format-upgrade
and soon realized that dealing with arbitrary changing service numbers
was a nightmare, and I could make my life a lot easier if the service
numbers were deterministic. But to ensure a proper upgrade, they also
needed to be deterministic *before* the upgrade - so a 0.3.0.1 release
was necessary, but the master branch was 0.4.0.0, so I created a
branch to work on 0.3.0.1.



 and the current upgrade notes don't even mention 0.3.0.1:


 That is my bad: I didn't backport the 0.3.0.1 lines to the master 
branch.

Thanks for the report, will fix.



So, in the end, does upgrading from 0.3.0.0 to 0.4.0.x require an
intermediate 0.3.0.1 s6-rc-compile + s6-rc-update step, and then
0.4.0.x s6-rc-compile + s6-rc-format-upgrade, or not?


 Yes, it does! You can only upgrade to 0.4.0.x from 0.3.0.1. Sorry
for not making it clearer, and sorry for the inconvenience; later
format upgrades will be simpler.



(I didn't have running services that I had to preserve, so personally
I didn't need any special care and just upgraded from whatever I had
at the time to s6-rc-0.4.0.0, but s6-rc-format-upgrade is now there
and one might need to explain or remind correct usage :) )


 If you don't mind breaking things and manually rebooting, you can
of course just yolo the db change without bothering with intermediary
steps. :)

--
 Laurent