Re: [Nix-dev] Hydra and security updates

[Profpatsch]

On 17-06-04 12:35am, Nicolas Pierron wrote:
> So currently, this project is held by a dead-lock between people
> asking me to demonstrate a large scale example, and having the
> infrastructure to doing so.

I think most of the lockup stems from people
not really knowing what your framework means in practice.

Would you be willing to do a talk about that
(and the remaining work tbd) on Nixcon?

-- 
Proudly written in Mutt with Vim on NixOS.
Q: Why is this email five sentences or less?
A: http://five.sentenc.es
May take up to five days to read your message. If it’s urgent, call me.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Nicolas Pierron]

On Sun, Jun 4, 2017 at 1:17 AM, Bjørn Forsman  wrote:
> On 4 June 2017 at 00:35, Nicolas Pierron  wrote:
>> So I started SOS [1] to make Nixpkgs more
>> declarative.  Thus removing some of the function overhead from
>> packages, which would help fixing a lot of the issues reported by the
>> static-analysis.
>
> I think you forgot to add the link to the SOS thing. (I'm interested.)
>
> Best regards,
> Bjørn Forsman

[1] https://github.com/NixOS/rfcs/pull/3

-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Bjørn Forsman]

On 4 June 2017 at 00:35, Nicolas Pierron  wrote:
> So I started SOS [1] to make Nixpkgs more
> declarative.  Thus removing some of the function overhead from
> packages, which would help fixing a lot of the issues reported by the
> static-analysis.

I think you forgot to add the link to the SOS thing. (I'm interested.)

Best regards,
Bjørn Forsman
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Nicolas Pierron]

On Sat, Jun 3, 2017 at 1:26 PM, Graham Christensen  wrote:
> This is part of my inclination of not really loving PR#10851, it is
> complicated and goes around the normal proceses, even when we can easily
> deploy fairly quickly.

The problem that I have with the current solutions is that they
involve _user_ actions (*), they do not address all uses cases, and
potentially a lot of local recompilation time.

PR#10851 goals are to address all of these issues.
>From the _maintainer_ point-of-view, this would be as simple as a
cherry-pick (*).
>From the user point of view, this would be like any ordinary channel.
>From hydra point of view, this would be only rebuilding packages which
have patches, or which are statically linked.

(*) goes around the normal processes.

-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Nicolas Pierron]

On Sat, Jun 3, 2017 at 12:54 AM, Leo Gaspard  wrote:
> On 06/02/2017 12:05 PM, Domen Kožar wrote:
>>> I see two ways of doing this: either having hydra somehow handle with
>>> special care security updates (hard to do)
>>
>> https://github.com/NixOS/nixpkgs/pull/10851
>
> This looks great!
>
> Unfortunately, it doesn't appear to be close to merging (esp. as it has
> merge conflicts), so I guess that's the best solution that isn't coming
> up right now? So having master and stable always build may be a current
> path forward, not yet as good as this PR but a good stop-gap.

I started a branch at the end of last year, which include these
changes and rebased them on top of the latest master, but I gave up as
I did not got any feedback for getting any Hydra infrastructure in
place to make use of this feature in a testing branch.  Having Hydra
infra in place would be among the next step to demonstrate the
usefulness of this approach, and convince more people to help fix the
static-analysis reports.

So currently, this project is held by a dead-lock between people
asking me to demonstrate a large scale example, and having the
infrastructure to doing so.

Most of the time, unpatched dependencies from PR#10851 are coming from
the fact that dependencies are resolved by functions them-self taken
for older generations of the fix-point, breaking the hypothesis on
which PR#10851 is based on. So I started SOS [1] to make Nixpkgs more
declarative.  Thus removing some of the function overhead from
packages, which would help fixing a lot of the issues reported by the
static-analysis.

-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Graham Christensen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


Leo Gaspard  writes:

> I just wanted to point out an issue with hydra: it doesn't make any
> distinction between security updates and normal changes.
>
> For example, [1] was released two days ago. Despite the fix landing two
> days ago too [2], nixos-unstable still doesn't have the vulnerability
> fixed.

nixos-unstable frequently lags behind for quite some time, and has no
guarantees about how quickly it'll receive security patches. You may be
interested in  nixos-unstable-small, which received the security update
much faster.

While it is fun and nice to think through various solutions to making
our unstable channel get security updates faster, I believe three
things that make it somewhat less critical:

1. The stable and ecommended version of NixOS to run is NixOS 17.03,
which also received the patch quite quickly.

2. There are strategies in place that can side-step the long rebuild
process if required, however they're typically not necessary. On a "the
world is burning" scale problem, nixos has seen a full rebuild from
nothing to channel published in 24 hours.

This is part of my inclination of not really loving PR#10851, it is
complicated and goes around the normal proceses, even when we can easily
deploy fairly quickly.

Most distributions have much more than 24 hours to be notified of an
issue and prepare a release, via the embargoed announcements on the
- -distro mailing list. Unfortunately that list is not accepting new
distro members at this time:
https://github.com/NixOS/nixpkgs/issues/14819

3. The much larger, more difficult problem is organizing _around_ the
security updates and getting them done regularly. These big scary bugs
are important yes, but so are the dozens of little bugs that get patched
weekly in various projects. Many of these are currently going unpatched.
For several months, I organized a weekly bug roundup that handled most
of these. When my bug source dried up, I decided to step away for a
time. I think I'm ready to start again, but need to do some research.

Regarding Hydra building PRs, that was an experiment to see how much
hardware and resources it would take. The integration with GitHub was
not as secure as we'd like, and wasn't suitable for merging with the
official hydra. There have been a few attempts at fixing it. If you'd
like to talk about it and take a crack, I'd be happy to talk with you!

Best,
Graham
-BEGIN PGP SIGNATURE-

iQIzBAEBCAAdFiEEP+htk0GpxXspt+y6BhIdNm/pQ1wFAlkynPoACgkQBhIdNm/p
Q1yIlw/8DkAEebiHjA+WCLDI6EIkqU0DW/xJDgklQOhILb6tyI/v9E5ip4yhHMrK
K9mjNexHTZSMLZJnFZExuznOAKFOro8YaflWu0RQl/gI3ZMXN+deTstM6S/ETFw6
5k4IYQVk/QBud3JpCUKgEPT1xi9q/CakNtdKMG7Mqxbvp1TljUwre8zk9qfHf1d1
mAWJC7Xhte3cuVzD5yMxnRJJVNhzxS1c7E2XSiSBlpJE3NZbBlr41CDTP63ASPIG
N/aslCw7Jj1RK6mxEHpWRXBQ8C88V17eUFrdB/pYggxmawhlQjSsEJSQ3DN4oib/
7bdvje0EGQGlusEycYQmDlVrMYrWSmKwKGqjF5oQgWxiYq9oTU5SU1dGfsFk8Xqc
DBOW1d2wc+9rdfuZbTbSaooJZOU5ACRyDEjxJYAqTdl4kbDXtGcQGUC14PFbGZWm
71Bl3bJE626Q2ioGPTBfhnmnqRcLkHX9kcYIFVV7G15zD23Ekf6VNHBdqnAf7szg
S0qriB+gh4fE8o63IhhCaTP0rwONZd7HoEVXCRa8FmkEypA+Vr9lCowBeik3DPHi
xSKTmOYg8Wr/RnemcwH1Jp1IFsGMy/ZgNKG9SqEv2PS8ocqPpK3j3QjBe0cw+Kyv
Jc9poZJOJdM8a6RxEn/Nq3Pd7bGod9AbP/O5OsE+60tnYLo30+4=
=XNPZ
-END PGP SIGNATURE-
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Danylo Hlynskyi]

So, the assumption is: "security updates hardly should break stuff, so we
can apply them without tests"
And desire is: "don't publish untested changes to channel"

This clearly leads to necessity of two channels, just as described in
https://github.com/NixOS/nixpkgs/pull/10851#issuecomment-212099317

The second channel, like `nixpkgs-secure`, shouldn't be a fork of nixpkgs
with
`quickfix`-es, but perhaps an overlay with security patches.
It is then included like `nixpkgs.overlays = [ (import
).channel-unstable ];`

This would require for user to "configure" security-update system, and
maintainers to
update nixpkgs-secure package database alongside with nixpkgs master and
nixpkgs stable.

--

Another option is to maintain branches with sufficies "-secure".

1. maintainers should add/remove security patches to "XXX-secure" branches
in nixpkgs-channels
2. hydra should regularly merge "XXX-secure" to "XXX"
3. hydra should merge "XXX" to "XXX-secure" on channel updates
4. hydra should build "XXX" and "XXX-secure" in parallel, and publish
channel whichever finishes faster


2017-06-03 4:13 GMT+03:00 Leo Gaspard :

> On 06/03/2017 01:55 AM, Frank wrote:
> > Op 3-6-2017 om 0:59 schreef Leo Gaspard:
> >> On 06/02/2017 06:54 PM, Frank wrote:
> >>> Op 1-6-2017 om 23:32 schreef Leo Gaspard:
>  Hi all,
> 
>  I just wanted to point out an issue with hydra: it doesn't make any
>  distinction between security updates and normal changes.
> >>> Why is this an issue? Security-updates are just as likely to introduce
> >>> bugs as every other update.
> >> If I have to choose between having a security vulnerability and having
> >> some installer tests that don't build (as these seem to be the source of
> >> most test failures)... I know what I'd rather have (especially given
> >> install images aren't generated from every commit of nixpkgs), don't you
> >> think?
> > You mean al the tests that didn't catch the bug in the first place? Or
> > the tests that assure the fix will be installed without problems?
> >
> > If the testing is a problem for distributing the software, the tests are
> > probably wrong. You can't fix things by testing, so don't try to repeat
> > and improve the upstream testing (not during distribution at least).
> >
> > The focus of the distribution is, distributing software, that installs
> > well on all target systems. And if your fix breaks some systems it
> > doesn't matter how important it is for security.
> >
> > I really agree, it's important to roll out security fixes fast. But I
> > don't see why other updates should be very time consuming.
>
> OK, I think I failed explaining what I think the issue is.
>
> In my mind, the issue is not having a security fix that breaks tests*,
> as fixes are(/should be) tested by upstream to not change any observable
> behavior except the actual security flaw.
>
> However, the issue is in having security fixes being delayed by
> unrelated commits that break tests. Because those other packages are way
> more disruptive than a security fix, and can (often) break tests, as
> there is no enforced "must pass tests on hydra" before merging a PR.
>
>
> * Even though I'd bet that may happen with transient test failures --
> and I'd still want that patch, so that anyone can't break in my system,
> even though it may mean some features not working perfectly as intended:
> time for tests is when preparing the patch, patching systems should be
> done within a few hours at most to consistently avoid attacks, and a few
> hours is hardly enough to even rebuild the system and get people to
> patch. Like, major distros get an ahead-of-time notification of serious
> flaws and prepare and pre-build the patch before it's even known to us,
> just to get the patch out faster... But it's not my main point, as this
> should actually just never happen, the choice of behavior in this case
> is irrelevant.
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Leo Gaspard]

On 06/03/2017 01:55 AM, Frank wrote:
> Op 3-6-2017 om 0:59 schreef Leo Gaspard:
>> On 06/02/2017 06:54 PM, Frank wrote:
>>> Op 1-6-2017 om 23:32 schreef Leo Gaspard:
 Hi all,

 I just wanted to point out an issue with hydra: it doesn't make any
 distinction between security updates and normal changes.
>>> Why is this an issue? Security-updates are just as likely to introduce
>>> bugs as every other update.
>> If I have to choose between having a security vulnerability and having
>> some installer tests that don't build (as these seem to be the source of
>> most test failures)... I know what I'd rather have (especially given
>> install images aren't generated from every commit of nixpkgs), don't you
>> think?
> You mean al the tests that didn't catch the bug in the first place? Or
> the tests that assure the fix will be installed without problems?
> 
> If the testing is a problem for distributing the software, the tests are
> probably wrong. You can't fix things by testing, so don't try to repeat
> and improve the upstream testing (not during distribution at least).
> 
> The focus of the distribution is, distributing software, that installs
> well on all target systems. And if your fix breaks some systems it
> doesn't matter how important it is for security.
> 
> I really agree, it's important to roll out security fixes fast. But I
> don't see why other updates should be very time consuming.

OK, I think I failed explaining what I think the issue is.

In my mind, the issue is not having a security fix that breaks tests*,
as fixes are(/should be) tested by upstream to not change any observable
behavior except the actual security flaw.

However, the issue is in having security fixes being delayed by
unrelated commits that break tests. Because those other packages are way
more disruptive than a security fix, and can (often) break tests, as
there is no enforced "must pass tests on hydra" before merging a PR.


* Even though I'd bet that may happen with transient test failures --
and I'd still want that patch, so that anyone can't break in my system,
even though it may mean some features not working perfectly as intended:
time for tests is when preparing the patch, patching systems should be
done within a few hours at most to consistently avoid attacks, and a few
hours is hardly enough to even rebuild the system and get people to
patch. Like, major distros get an ahead-of-time notification of serious
flaws and prepare and pre-build the patch before it's even known to us,
just to get the patch out faster... But it's not my main point, as this
should actually just never happen, the choice of behavior in this case
is irrelevant.



signature.asc
Description: OpenPGP digital signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Frank]

Op 3-6-2017 om 0:59 schreef Leo Gaspard:

On 06/02/2017 06:54 PM, Frank wrote:

Op 1-6-2017 om 23:32 schreef Leo Gaspard:

Hi all,

I just wanted to point out an issue with hydra: it doesn't make any
distinction between security updates and normal changes.

Why is this an issue? Security-updates are just as likely to introduce
bugs as every other update.

If I have to choose between having a security vulnerability and having
some installer tests that don't build (as these seem to be the source of
most test failures)... I know what I'd rather have (especially given
install images aren't generated from every commit of nixpkgs), don't you
think?
You mean al the tests that didn't catch the bug in the first place? Or 
the tests that assure the fix will be installed without problems?


If the testing is a problem for distributing the software, the tests are 
probably wrong. You can't fix things by testing, so don't try to repeat 
and improve the upstream testing (not during distribution at least).


The focus of the distribution is, distributing software, that installs 
well on all target systems. And if your fix breaks some systems it 
doesn't matter how important it is for security.


I really agree, it's important to roll out security fixes fast. But I 
don't see why other updates should be very time consuming.


Greetings,
Frank
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Leo Gaspard]

On 06/02/2017 12:05 PM, Domen Kožar wrote:
>> I see two ways of doing this: either having hydra somehow handle with
>> special care security updates (hard to do)
> 
> https://github.com/NixOS/nixpkgs/pull/10851

This looks great!

Unfortunately, it doesn't appear to be close to merging (esp. as it has
merge conflicts), so I guess that's the best solution that isn't coming
up right now? So having master and stable always build may be a current
path forward, not yet as good as this PR but a good stop-gap.

>> , or having master and stable branches *always* build.
> 
> For that we'd need to have infrastructure that builds PRs and reports
> status on github.
> 
> I think that's the right solution, but it does involve work and Hydra
> improvements :)

Hmm, "[Nix-dev] Hydra Building PRs" (2017-01-28) made me think hydra was
actually ready and the only needed thing was setting it up on
hydra.nixos.org and use a github bot?

I mean, if it's waiting for only a github bot then I guess it'll be easy
to do (and the fact at least two persons said they were doing it 3
months ago makes me think it's actually already ready) but the part
about setting it up at hydra.nixos.org can be made only by a select few
people, so I guess there is not much most of us can do about it?



signature.asc
Description: OpenPGP digital signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Frank]

Op 1-6-2017 om 23:32 schreef Leo Gaspard:

Hi all,

I just wanted to point out an issue with hydra: it doesn't make any
distinction between security updates and normal changes.


Why is this an issue? Security-updates are just as likely to introduce 
bugs as every other update.


Greetings,
Frank
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[Domen Kožar]

> I see two ways of doing this: either having hydra somehow handle with
> special care security updates (hard to do)

https://github.com/NixOS/nixpkgs/pull/10851

> , or having master and stable branches *always* build.

For that we'd need to have infrastructure that builds PRs and reports
status on github.

I think that's the right solution, but it does involve work and Hydra
improvements :)

On Thu, Jun 1, 2017 at 11:53 PM,  wrote:

> On Thu, Jun 1, 2017, at 23:32, Leo Gaspard wrote:
> > Hi all,
> >
> > [ ... ]
>
> I think this is relevant to your interests:
> https://github.com/NixOS/nixpkgs/pull/10851
>
> On a side note, I don't know why anybody would actually run
> nixos-unstable; it gets stuck for long periods of time quite often ... I
> think sticking to the latest release channel or using the -small variant
> is better, depending on whether you want/need the latest bugs.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Hydra and security updates

[joachifm]

On Thu, Jun 1, 2017, at 23:32, Leo Gaspard wrote:
> Hi all,
> 
> [ ... ]

I think this is relevant to your interests:
https://github.com/NixOS/nixpkgs/pull/10851

On a side note, I don't know why anybody would actually run
nixos-unstable; it gets stuck for long periods of time quite often ... I
think sticking to the latest release channel or using the -small variant
is better, depending on whether you want/need the latest bugs.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev