Re: [gentoo-dev] Signed push & clock drift rejection

[Alec Warner]

On Tue, Jul 19, 2016 at 11:23 AM, Andrew Savchenko 
wrote:

> On Mon, 18 Jul 2016 22:21:22 -0400 waltd...@waltdnes.org wrote:
> > On Mon, Jul 18, 2016 at 11:27:09PM +0300, Andrew Savchenko wrote
> > >
> > > As I wrote earlier in this thread, ntp server is not a guarantee
> > > that such problems will not happen. If hardware clocked was
> > > significantly offset during boot, it may take several _hours_ for
> > > ntp to fix this via clock skew. Apparantly commit may be made
> > > during these several hours.
> >
> >   I'm amazed that "robust linux servers" are deathly afraid of simply
> > setting the time, and being done with it. And while we're at it, if a
> > developer is doing development on a server machine, he may have other
> > problems to worry about.  At home I occasionally manually run a script
> > that includes the 2 lines...
>

Its actually really typical for programmers to do bad things with time.
Some examples include:

1) Assuming time is monotonically increasing.
2) Assuming time changes at a constant rate.

Doing 1 leads to bugs where your local clock is faster than 'real time' and
your time sync daemon sets your time to a time in the past. This is a very
bad thing for a lot of workloads. Generally callers should use
CLOCK_MONOTONIC (or CLOCK_MONOTONIC_RAW) to properly use a clock, but many
do not.

Doing 2 leads to synchronization issues because you expected APIs like:

DoThing()
WaitFiveSeconds() # Actually expect 5 seconds to pass here?, haha!
DoOtherThing()

Of course this is also a bad pattern, but you'd be surprised where you see
this crop up.

Not to mention other workloads where time is part the security stack
(generating randomness, GUIDs, nonces, etc, avoiding replay, etc.)

-A


> I never said anything about "robust linux servers". If you think
> than only servers need a gradual clock slew instead of stepping,
> you are mistaken.
>
> > /usr/bin/sudo /usr/bin/openrdate -n -s ca.pool.ntp.org
> > /usr/bin/sudo /sbin/hwclock --systohc
>
> And if time delta is significant, the system may become broken
> this way. Congratulations.
>
> The gradual NTP time slew was not invented because people were lazy
> to run two simple commands. Actually it is just the opposite:
> setting system time via a single huge step is much easier to
> implement than a proper adjustment via a series of small time slews.
> For servers it is indeed important in many ways, including
> time stamp based cryptography as kerberos or database integrity.
>
> But desktops also do need a proper time adjustment:
> - Filesystems will not operate correctly with time stamps in
> future, in the best keys they will be marked/reported as needed a
> repair procedure.
> - Cron jobs may go broken too as chithanh mentioned already.
> - Video encoding is not happy with time shifts at all. While small
> predictable slews can be tolerated, large jumps will just broke the
> process.
> - System may become *vulnerable* because of time stamp based attack.
> Though it is not easy to use such behaviour, it is still possible.
> - and many more...
>
> Best regards,
> Andrew Savchenko
>

Re: [gentoo-dev] Signed push & clock drift rejection

[Patrick McLean]

On Tue, 19 Jul 2016 13:22:29 -0700
Patrick McLean  wrote:

> On Tue, 19 Jul 2016 21:23:16 +0300
> Andrew Savchenko  wrote:
> > On Mon, 18 Jul 2016 22:21:22 -0400 waltd...@waltdnes.org wrote:  
> > > On Mon, Jul 18, 2016 at 11:27:09PM +0300, Andrew Savchenko
> > > wrote
> > > > 
> > > > As I wrote earlier in this thread, ntp server is not a guarantee
> > > > that such problems will not happen. If hardware clocked was
> > > > significantly offset during boot, it may take several _hours_
> > > > for ntp to fix this via clock skew. Apparantly commit may be
> > > > made during these several hours.
> > > 
> > >   I'm amazed that "robust linux servers" are deathly afraid of
> > > simply setting the time, and being done with it. And while we're
> > > at it, if a developer is doing development on a server machine,
> > > he may have other problems to worry about.  At home I occasionally
> > > manually run a script that includes the 2 lines...
> > 
> > I never said anything about "robust linux servers". If you think
> > than only servers need a gradual clock slew instead of stepping,
> > you are mistaken.
> >   
> > > /usr/bin/sudo /usr/bin/openrdate -n -s ca.pool.ntp.org
> > > /usr/bin/sudo /sbin/hwclock --systohc
> > 
> > And if time delta is significant, the system may become broken
> > this way. Congratulations.  
> 
> Really? I have many times skewed time by weeks using ntpdate with no
> issues.
> 
> > The gradual NTP time slew was not invented because people were lazy
> > to run two simple commands. Actually it is just the opposite:
> > setting system time via a single huge step is much easier to
> > implement than a proper adjustment via a series of small time slews.
> > For servers it is indeed important in many ways, including
> > time stamp based cryptography as kerberos or database integrity.  
> 
> Sure randomly skewing time can cause weird issues, which is why it is
> a manual operation (and/or boot time operation).
> 
> > 
> > But desktops also do need a proper time adjustment:
> > - Filesystems will not operate correctly with time stamps in
> > future, in the best keys they will be marked/reported as needed a
> > repair procedure.  
> 
> I have only ever seen ext4 complain about the superblock timestamp,
> and fsck generally just corrects without issue, and it generally is
> only an issue after a reboot.
> 
> > - Cron jobs may go broken too as chithanh mentioned already.  
> 
> Get a better crond? Decent cron daemons can detect time skews and act
> accordingly.
> 
> > - Video encoding is not happy with time shifts at all. While small
> > predictable slews can be tolerated, large jumps will just broke the
> > process.  
> 
> Anything that cares about having a monotonic clock, and doesn't
> actually care about the real time (like video encoding) should just
> use the monotonic clock, not the system time.
> 
> > - System may become *vulnerable* because of time stamp based attack.
> > Though it is not easy to use such behaviour, it is still possible.  
> 
> Once again, monotonic clock exists for a reason. If you want to
> talk about vulnerabilities, you do realize that doesn't work properly
> unless the client and server have reasonably similar system times.
> 
Err, SSL doesn't work properly.

> > - and many more...
> > 
> > Best regards,
> > Andrew Savchenko  
> 
> 

Re: [gentoo-dev] Signed push & clock drift rejection

[Alec Warner]

On Tue, Jul 19, 2016 at 4:31 AM, Consus  wrote:

> On 18:03 Fri 15 Jul, Robin H. Johnson wrote:
> > Hi all,
> >
> > In tracing down problems with the git->rsync path, it has been noticed
> > that some developers have significant clock drift on their local systems
> > (up to one case of 14 days wrong), and it's potentially contributing to
> > problems in generating the rsync tree.
> >
> > I have implemented a check as part of the hook that validates Git push
> > certificates (require-signed-push). It looks for clock drift or an
> > overly long push, and aborts if needed.
> >
> > The tolerances are presently set to:
> > - 5 seconds of clock drift.
> > - 'git push' must be completed in 60 seconds.
>
> AFAIR Windows AD servers allow 5 minutes clock drift. How about at least
> a minute or so?
>
>
I would also argue for 5 minutes.

-A

Re: [gentoo-dev] Signed push & clock drift rejection

[Patrick McLean]

On Tue, 19 Jul 2016 21:23:16 +0300
Andrew Savchenko  wrote:
> On Mon, 18 Jul 2016 22:21:22 -0400 waltd...@waltdnes.org wrote:
> > On Mon, Jul 18, 2016 at 11:27:09PM +0300, Andrew Savchenko wrote  
> > > 
> > > As I wrote earlier in this thread, ntp server is not a guarantee
> > > that such problems will not happen. If hardware clocked was
> > > significantly offset during boot, it may take several _hours_ for
> > > ntp to fix this via clock skew. Apparantly commit may be made
> > > during these several hours.  
> > 
> >   I'm amazed that "robust linux servers" are deathly afraid of
> > simply setting the time, and being done with it. And while we're at
> > it, if a developer is doing development on a server machine, he may
> > have other problems to worry about.  At home I occasionally
> > manually run a script that includes the 2 lines...  
> 
> I never said anything about "robust linux servers". If you think
> than only servers need a gradual clock slew instead of stepping,
> you are mistaken.
> 
> > /usr/bin/sudo /usr/bin/openrdate -n -s ca.pool.ntp.org
> > /usr/bin/sudo /sbin/hwclock --systohc  
> 
> And if time delta is significant, the system may become broken
> this way. Congratulations.

Really? I have many times skewed time by weeks using ntpdate with no
issues.

> The gradual NTP time slew was not invented because people were lazy
> to run two simple commands. Actually it is just the opposite:
> setting system time via a single huge step is much easier to
> implement than a proper adjustment via a series of small time slews.
> For servers it is indeed important in many ways, including
> time stamp based cryptography as kerberos or database integrity.

Sure randomly skewing time can cause weird issues, which is why it is a
manual operation (and/or boot time operation).

> 
> But desktops also do need a proper time adjustment:
> - Filesystems will not operate correctly with time stamps in
> future, in the best keys they will be marked/reported as needed a
> repair procedure.

I have only ever seen ext4 complain about the superblock timestamp,
and fsck generally just corrects without issue, and it generally is
only an issue after a reboot.

> - Cron jobs may go broken too as chithanh mentioned already.

Get a better crond? Decent cron daemons can detect time skews and act
accordingly.

> - Video encoding is not happy with time shifts at all. While small
> predictable slews can be tolerated, large jumps will just broke the
> process.

Anything that cares about having a monotonic clock, and doesn't
actually care about the real time (like video encoding) should just use
the monotonic clock, not the system time.

> - System may become *vulnerable* because of time stamp based attack.
> Though it is not easy to use such behaviour, it is still possible.

Once again, monotonic clock exists for a reason. If you want to
talk about vulnerabilities, you do realize that doesn't work properly
unless the client and server have reasonably similar system times.

> - and many more...
> 
> Best regards,
> Andrew Savchenko

Re: [gentoo-dev] Signed push & clock drift rejection

[R0b0t1]

On Tue, Jul 19, 2016 at 1:23 PM, Andrew Savchenko  wrote:
> - System may become *vulnerable* because of time stamp based attack.
> Though it is not easy to use such behaviour, it is still possible.

All offline attacks are harder, even ones which haven't been invented yet.

Re: [gentoo-dev] Signed push & clock drift rejection

[Andrew Savchenko]

On Mon, 18 Jul 2016 22:21:22 -0400 waltd...@waltdnes.org wrote:
> On Mon, Jul 18, 2016 at 11:27:09PM +0300, Andrew Savchenko wrote
> > 
> > As I wrote earlier in this thread, ntp server is not a guarantee
> > that such problems will not happen. If hardware clocked was
> > significantly offset during boot, it may take several _hours_ for
> > ntp to fix this via clock skew. Apparantly commit may be made
> > during these several hours.
> 
>   I'm amazed that "robust linux servers" are deathly afraid of simply
> setting the time, and being done with it. And while we're at it, if a
> developer is doing development on a server machine, he may have other
> problems to worry about.  At home I occasionally manually run a script
> that includes the 2 lines...

I never said anything about "robust linux servers". If you think
than only servers need a gradual clock slew instead of stepping,
you are mistaken.

> /usr/bin/sudo /usr/bin/openrdate -n -s ca.pool.ntp.org
> /usr/bin/sudo /sbin/hwclock --systohc

And if time delta is significant, the system may become broken
this way. Congratulations.

The gradual NTP time slew was not invented because people were lazy
to run two simple commands. Actually it is just the opposite:
setting system time via a single huge step is much easier to
implement than a proper adjustment via a series of small time slews.
For servers it is indeed important in many ways, including
time stamp based cryptography as kerberos or database integrity.

But desktops also do need a proper time adjustment:
- Filesystems will not operate correctly with time stamps in
future, in the best keys they will be marked/reported as needed a
repair procedure.
- Cron jobs may go broken too as chithanh mentioned already.
- Video encoding is not happy with time shifts at all. While small
predictable slews can be tolerated, large jumps will just broke the
process.
- System may become *vulnerable* because of time stamp based attack.
Though it is not easy to use such behaviour, it is still possible.
- and many more...

Best regards,
Andrew Savchenko


pgpTgEx3U6z6j.pgp
Description: PGP signature

Re: [gentoo-dev] Signed push & clock drift rejection

[Andrew Savchenko]

Hi,

On Mon, 18 Jul 2016 17:20:50 -0400 Rich Freeman wrote:
> On Sat, Jul 16, 2016 at 5:33 AM, Andrew Savchenko  wrote:
> >
> > On Fri, 15 Jul 2016 18:03:30 + Robin H. Johnson wrote:
> >>
> >> The tolerances are presently set to:
> >> - 5 seconds of clock drift.
> >
> > Set it for a minute or two. This will protect from commits from
> > really out-of-sync systems (like 14 days mentioned above) and will
> > keep usablity hight for others.
> 
> I'll defer to infra on how much they can accept, but I tend to think
> that we can afford to be a bit more liberal.  However, I don't think
> we want to accept things like systems coming out of suspend that are
> off by hours.

Nobody asks for hours :) 5 minutes should be fine and sane.

> >
> >> - 'git push' must be completed in 60 seconds.
> >
> > Why?! What is wrong if push will take 120 seconds? I often commit
> > from quite an old box and git push takes 20-40 seconds, while this
> > is within your limits, the margin is not safe.
> >
> > What if someone needs to commit via 2G GPRS or similar slow network
> > link? Afaik we have developers on quite slow and unstable links.
> >
> > Just set this limit to 5 minutes to make it a sane protection of a
> > stale push.
> >
> 
> Somebody can correct me if I'm wrong, but I'm pretty sure that only
> one person can be pushing anything at time.

Indeed so.

> So, regardless of any
> rsync limitations, I'm not sure we really want developers to be
> spending 5 minutes doing a push.  That means that if anybody else does
> a commit during that 5 minutes they're going to have to rebase it.

Let us consider which way we'll make more harm than good.

If git push takes longer than a minute due to a slow box or a bad
uplink, there is no way to fix this. If current limit will be
enforced further, we'll actually ban developers with slow
environment from contributing to Gentoo.

As for rebasing, this is simple and can be automated by
pull.rebase=true (it should be disabled during merge commits and
can be done so on the command line of course, but random merge
commits are not welcomed due to our git workflow policy, so this is
a rare event).

If I understand a reasoning behind the git push time constraint
correctly, it is needed as a safeguard from stale pushes, so events
where push takes > 1 minute should be very rare, thus a little harm
can be done by raising git push limit to 5 minutes.

Best regards,
Andrew Savchenko


pgpyDRzcw6LFA.pgp
Description: PGP signature

Re: [gentoo-dev] Signed push & clock drift rejection

[Michał Górny]

On Mon, 18 Jul 2016 22:03:35 +0200
Marc Schiffbauer  wrote:

> * Rafael Goncalves Martins schrieb am 18.07.16 um 03:12 Uhr:
> > On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko  
> > wrote:  
> > > Set it for a minute or two. This will protect from commits from
> > > really out-of-sync systems (like 14 days mentioned above) and will
> > > keep usablity hight for others.  
> > 
> > I second this "request" :)
> > 
> > remote: Your system clock is off by 6 seconds (limit 5)  
> 
> Why not fix your system clock? No ntpd running?
> 
> Check 'ntpq -pn'

But Gentoo is about choice! Developers should have a choice! If they
want to use systems with completely broken clock, why do you want to
take away that choice from them? They have a right to push commits
with completely useless timestamps if they desire, or to break rsync
generation for our users every now and then!

-- 
Best regards,
Michał Górny



pgpyCQ7o4UKyB.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Signed push & clock drift rejection

[Chí-Thanh Christopher Nguyễn]

waltd...@waltdnes.org schrieb:

And even if the system is behind time, it can cause problems. cronjobs
running unexpectedly close to each other (or missed cronjobs in extreme
cases). User sessions expiring early, etc.

And even if there is only one second, and that is known well ahead
(e.g. leap seconds): Unless you know that there isn't going to be
a problem, a great deal of care needs to go into handling that.

   In that case, the dev machine should be a separate machine from the
server.  They don't even have to be separate physical machines.  Do the
dev work in a VM, and set the time in the VM just before doing the push.


I am not talking about bircoph's dev machine, I am explaining why ntpd 
works the way it does. Which usually doesn't lead to problems as the 
drift will be corrected sooner or later. So there was never a problem 
nor reason for action in either case, until it was decided that 5 
seconds is the maximum drift for a push to gentoo.git.



Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] Signed push & clock drift rejection

[Consus]

On 18:03 Fri 15 Jul, Robin H. Johnson wrote:
> Hi all,
> 
> In tracing down problems with the git->rsync path, it has been noticed
> that some developers have significant clock drift on their local systems
> (up to one case of 14 days wrong), and it's potentially contributing to
> problems in generating the rsync tree.
> 
> I have implemented a check as part of the hook that validates Git push
> certificates (require-signed-push). It looks for clock drift or an
> overly long push, and aborts if needed.
> 
> The tolerances are presently set to:
> - 5 seconds of clock drift.
> - 'git push' must be completed in 60 seconds.

AFAIR Windows AD servers allow 5 minutes clock drift. How about at least
a minute or so?

Re: [gentoo-dev] Signed push & clock drift rejection

[waltdnes]

On Tue, Jul 19, 2016 at 10:07:12AM +0200, Chí-Thanh Christopher Nguy???n wrote
> Kent Fredric schrieb:
> > On Mon, 18 Jul 2016 22:21:22 -0400
> > waltd...@waltdnes.org wrote:
> >
> >>I'm amazed that "robust linux servers" are deathly afraid of simply
> >> setting the time, and being done with it.
> >
> > There's problems at the software level everywhere that are not so
> > simply solved.
> >
> > A more obvious example is in the event your system time gets *ahead* of
> > real-time.
> 
> And even if the system is behind time, it can cause problems. cronjobs 
> running unexpectedly close to each other (or missed cronjobs in extreme 
> cases). User sessions expiring early, etc.
> 
> And even if there is only one second, and that is known well ahead
> (e.g. leap seconds): Unless you know that there isn't going to be
> a problem, a great deal of care needs to go into handling that.

  In that case, the dev machine should be a separate machine from the
server.  They don't even have to be separate physical machines.  Do the
dev work in a VM, and set the time in the VM just before doing the push.

-- 
Walter Dnes 
I don't run "desktop environments"; I run useful applications

Re: [gentoo-dev] Signed push & clock drift rejection

[Andrew Savchenko]

On Mon, 18 Jul 2016 16:25:34 -0500 james wrote:
> On 07/18/2016 03:03 PM, Marc Schiffbauer wrote:
> > * Rafael Goncalves Martins schrieb am 18.07.16 um 03:12 Uhr:
> >> On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko  
> >> wrote:
> >>> Set it for a minute or two. This will protect from commits from
> >>> really out-of-sync systems (like 14 days mentioned above) and will
> >>> keep usablity hight for others.
> >>
> >> I second this "request" :)
> >>
> >> remote: Your system clock is off by 6 seconds (limit 5)
> >
> > Why not fix your system clock? No ntpd running?
> >
> > Check 'ntpq -pn'
> >
> > -Marc
> >
> 
> net-misc/openntpd is simple and might do the job well enough, or is
> net-misc/ntp a hard requirement ?
> 
> I just use the  default (gentoo) time servers, for now, but perhaps 
> using specified servers in different regions might work too?

Any ntp sync daemon will be fine.

I prefer net-misc/chrony: it is simpler and more secure than ntpd
(e.g. look at CVE counts) and handles well situations when upstream
servers go offline (e.g. continues to apply calculated drift to rtc
clock). Though it doesn't support all features of ntpd (e.g. ssl),
it has all subset of them sufficient IRL.

Best regards,
Andrew Savchenko


pgpmupo_hke3F.pgp
Description: PGP signature

Re: [gentoo-dev] Signed push & clock drift rejection

[Chí-Thanh Christopher Nguyễn]

Kent Fredric schrieb:

On Mon, 18 Jul 2016 22:21:22 -0400
waltd...@waltdnes.org wrote:


   I'm amazed that "robust linux servers" are deathly afraid of simply
setting the time, and being done with it.


There's problems at the software level everywhere that are not so
simply solved.

A more obvious example is in the event your system time gets *ahead* of
real-time.


And even if the system is behind time, it can cause problems. cronjobs 
running unexpectedly close to each other (or missed cronjobs in extreme 
cases). User sessions expiring early, etc.


And even if there is only one second, and that is known well ahead (e.g. leap 
seconds): Unless you know that there isn't going to be a problem, a great 
deal of care needs to go into handling that.



Best regards,
Chí-Thanh Christopher Nguyễn




signature.asc
Description: OpenPGP digital signature