Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-12-13 Thread Ryan Harper 
On Tue, Dec 13, 2016 at 10:02 AM, Martin Pitt 
wrote:

> Ryan Harper [2016-12-06 12:54 -]:
> > The following change should go against systemd-networkd-wait-
> > online.service
> >
> > + # Ensure that DNS is working before reaching online target
> > + After=systemd-networkd-resolvconf-update.service
>
> For the record, this should be the other way around -- add
> Before=systemd-networkd-wait-online.service to
> s-n-resolvconf-update.service. The latter is a Debian downstream unit
> and thus avoids carrying a patch to an upstream unit that refers to a
> downstream one.
>

Well, ideally we'd have both.  Part of the challenge in dealing with
systemd units is that it's very difficult to determine the ordering.
If one doesn't look at the right file.

I won't push for a delta but I do think that these unit relationships ought
to be explicit on both sides.


>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1636912
>
> Title:
>   systemd-networkd runs too late for cloud-init.service (net)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1636912/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to resolvconf in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  Fix Released
Status in cloud-init package in Ubuntu:
  Triaged
Status in resolvconf package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Fix Released
Status in cloud-init source package in Xenial:
  Confirmed
Status in resolvconf source package in Xenial:
  In Progress
Status in systemd source package in Xenial:
  Fix Committed
Status in cloud-init source package in Yakkety:
  New
Status in resolvconf source package in Yakkety:
  In Progress
Status in systemd source package in Yakkety:
  Fix Committed
Status in resolvconf package in Debian:
  New

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
    └─basic.target @11.403s
  └─sockets.target @11.401s
    └─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
    └─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
    └─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
    └─run-cgmanager-fs.mount @12.687s
  └─local-fs-pre.target @1.393s
    └─systemd-tmpfiles-setup-dev.service @1.116s +195ms
  └─kmod-static-nodes.service @887ms +193ms
    └─system.slice @783ms
  └─-.slice @721ms

  cloud-init would need networkd to run at or before
  'networking.service' so it can raise networking to then find and use
  network-based datasources.

  # grep

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-12-13 Thread Martin Pitt 
Ryan Harper [2016-12-06 12:54 -]:
> The following change should go against systemd-networkd-wait-
> online.service
> 
> + # Ensure that DNS is working before reaching online target
> + After=systemd-networkd-resolvconf-update.service

For the record, this should be the other way around -- add
Before=systemd-networkd-wait-online.service to
s-n-resolvconf-update.service. The latter is a Debian downstream unit
and thus avoids carrying a patch to an upstream unit that refers to a
downstream one.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  Fix Released
Status in cloud-init package in Ubuntu:
  Triaged
Status in resolvconf package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Fix Released
Status in cloud-init source package in Xenial:
  Confirmed
Status in resolvconf source package in Xenial:
  In Progress
Status in systemd source package in Xenial:
  Fix Committed
Status in cloud-init source package in Yakkety:
  New
Status in resolvconf source package in Yakkety:
  In Progress
Status in systemd source package in Yakkety:
  Fix Committed
Status in resolvconf package in Debian:
  New

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
    └─basic.target @11.403s
  └─sockets.target @11.401s
    └─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
    └─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
    └─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
    └─run-cgmanager-fs.mount @12.687s
  └─local-fs-pre.target @1.393s
    └─systemd-tmpfiles-setup-dev.service @1.116s +195ms
  └─kmod-static-nodes.service @887ms +193ms
    └─system.slice @783ms
  └─-.slice @721ms

  cloud-init would need networkd to run at or before
  'networking.service' so it can raise networking to then find and use
  network-based datasources.

  # grep systemd /usr/share/snappy/dpkg.list
  ii  libnss-resolve:amd64  229-4ubuntu11   
 amd64nss module to resolve names via systemd-resolved
  ii  libpam-systemd:amd64  229-4ubuntu11   
 amd64system and service manager - PAM module
  ii  libsystemd0:amd64 229-4ubuntu11   
 amd64systemd utility library
  ii  systemd   229-4ubuntu11   
 amd64system and service manager
  ii  systemd-sysv  229-4ubuntu11   
 amd64system and

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-29 Thread Martin Pitt 
Steve Langasek [2016-10-29  4:38 -]:
> cloud-init does not *require* you to be online.  It *requires* you to
> provide a data source

Yes, I know, that's exactly my point -- by design it should/does not
require you to be online, but by making it wait for
s-n-wait-online.target or network-online.target you would introduces
this requirement. Hence my suggestion to dynamically call the wait
binaries when appropriate.

(I think we actually agree, and just talk past each other ☺ )

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  New
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
└─basic.target @11.403s
  └─sockets.target @11.401s
└─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
└─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
└─run-cgmanager-fs.mount @12.687s
  └─local-fs-pre.target @1.393s
└─systemd-tmpfiles-setup-dev.service @1.116s +195ms
  └─kmod-static-nodes.service @887ms +193ms
└─system.slice @783ms
  └─-.slice @721ms

  
  cloud-init would need networkd to run at or before 'networking.service' so it 
can raise networking to then find and use network-based datasources.

  # grep systemd /usr/share/snappy/dpkg.list 
  ii  libnss-resolve:amd64  229-4ubuntu11   
 amd64nss module to resolve names via systemd-resolved
  ii  libpam-systemd:amd64  229-4ubuntu11   
 amd64system and service manager - PAM module
  ii  libsystemd0:amd64 229-4ubuntu11   
 amd64systemd utility library
  ii  systemd   229-4ubuntu11   
 amd64system and service manager
  ii  systemd-sysv  229-4ubuntu11   
 amd64system and service manager - SysV links

  # grep cloud-init /usr/share/snappy/dpkg.list 
  ii  cloud-init
0.7.8-201610260005-gf7a5756-0ubuntu1~trunk~ubuntu16.04.1 all  Init 
scripts for cloud instances

To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/1636912/+subscriptions

-- 
Mailing list:

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-29 Thread Martin Pitt 
Ryan Harper [2016-10-28 18:46 -]:
> netplan generator only emits the "systemd-networkd" target wants, so
> if we use After=systemd-networkd-wait-online; that's never run since
> nothing wants it.

Right, if you want it, you need to pull it in yourself.

> If we add it explicitly, then it runs even when networkd doesn't

Oh, indeed -- it quickly ends with "Dependency failed" as networkd is
just a Requisite=, not a Requires=. So this actually does give you
exactly the semantics that we want, no? *If* networkd is running then
it waits for it, otherwise it's a no-op.

@Steve: ^ So I take back my previous comment -- it seems
Requires/After=s-n-wait-online.service is actually exactly what we
want after all.

> I did play with it, but the networkd in xenial blocks for some non-trivial
> amount of time (10s of seconds)
> if dbus.service is not up.

OK, that's something to look into then. It doesn't hang if there's no
configuration (i. e not trivially reproducible), I'll check if I can
reproduce this with a real config ASAP.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  New
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
└─basic.target @11.403s
  └─sockets.target @11.401s
└─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
└─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
└─run-cgmanager-fs.mount @12.687s
  └─local-fs-pre.target @1.393s
└─systemd-tmpfiles-setup-dev.service @1.116s +195ms
  └─kmod-static-nodes.service @887ms +193ms
└─system.slice @783ms
  └─-.slice @721ms

  
  cloud-init would need networkd to run at or before 'networking.service' so it 
can raise networking to then find and use network-based datasources.

  # grep systemd /usr/share/snappy/dpkg.list 
  ii  libnss-resolve:amd64  229-4ubuntu11   
 amd64nss module to resolve names via systemd-resolved
  ii  libpam-systemd:amd64  229-4ubuntu11   
 amd64system and service manager - PAM module
  ii  libsystemd0:amd64 229-4ubuntu11   
 amd64systemd utility library
  ii  systemd   229-4ubuntu11

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-28 Thread Steve Langasek 
On Fri, Oct 28, 2016 at 06:46:07PM -, Ryan Harper wrote:
> > > we really want something like
> > >  After=networking|networkd-wait-online
> > > which handles determining if networkd was supposed to run or not

> > That already exists, it's network-online.target -- whatever "implements"
> > it (ifupdown, networkd, NM) will hook itself into this target. Nothing
> > more, nothing less, so if cloud-init just wants to wait until it's
> > online, then just make it Requires/After=network-online.target instead
> > of Before= it. (But again -- this is a very strong dependency which is
> > very inconvenient anywhere but cloud environments with essentially one
> > virtual ethernet card).

> It may be that network-online.target is the right place.  Scott had some
> reason for not using that explicitly before; I expect some details from
> him.

Because network-online.target is not guaranteed to be reached for the
reasons Martin mentions, so cloud-init depending on it will forever block
other services from starting up, /even if/ you had a valid local data
source.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  New
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
└─basic.target @11.403s
  └─sockets.target @11.401s
└─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
└─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
└─run-cgmanager-fs.mount @12.687s
  └─local-fs-pre.target @1.393s
└─systemd-tmpfiles-setup-dev.service @1.116s +195ms
  └─kmod-static-nodes.service @887ms +193ms
└─system.slice @783ms
  └─-.slice @721ms

  
  cloud-init would need networkd to run at or before 'networking.service' so it 
can raise networking to then find and use network-based datasources.

  # grep systemd /usr/share/snappy/dpkg.list 
  ii  libnss-resolve:amd64  229-4ubuntu11   
 amd64nss module to resolve names via systemd-resolved
  ii  libpam-systemd:amd64  229-4ubuntu11   
 amd64system and service manager - PAM module
  ii  libsystemd0:amd64 229-4ubuntu11   
 amd64systemd utility library
  ii  systemd

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-28 Thread Steve Langasek 
On Fri, Oct 28, 2016 at 06:02:29PM -, Martin Pitt wrote:
> > However, if there isn't a local seed, then we must search again *once*
> networking is up.

> But this "if there isn't a local seed" isn't something you can express as
> a static condition, hence my thought that it might be better if c-i calls
> s-n-wait-online if and only if it's necessary.  But YMMV.

Yes, it's not a static condition; and even if it *were*, cloud-init should
apply sensible timeouts in the event that no network source is available.
So s-n-wait-online is still the better answer.

> > This works just fine with 'networking.service'

> This did/does not really work "fine" IMHO -- all of our cloud images
> hang for a long time at boot unless you give them a local data source or
> disable cloud-init.

That is the image working as designed, when booted in an environment it's
not designed for.

> It also imposes the restriction that you must be online during boot, which
> is fine for a cloud environment, but rather unfriendly for other
> scenarios.

cloud-init does not *require* you to be online.  It *requires* you to
provide a data source; as you've already pointed out, it can be a local disk
or it can be a network source.  Sometimes you have a disk source, sometimes
you have a network source; this is not a design decision of cloud-init, it's
a function of the *cloud environment* where you're booting the image, and
it's out of the scope of this bug to redesign cloud-init to be something
other than it is - a tool for provisioning generic images when booting
noninteractively in a cloud.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  New
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
└─basic.target @11.403s
  └─sockets.target @11.401s
└─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
└─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
└─run-cgmanager-fs.mount @12.687s
  └─local-fs-pre.target @1.393s
└─systemd-tmpfiles-setup-dev.service @1.116s +195ms
  └─kmod-static-nodes.service @887ms +193ms
└─system.slice @783ms
  └─-.slice @721ms

  
  cloud-init would need networkd to run at or before 'networking.service' so it 
can raise networking to then find and use network-based datasources.

  # grep

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-28 Thread Ryan Harper 
On Fri, Oct 28, 2016 at 1:02 PM, Martin Pitt 
wrote:

> > However, if there isn't a local seed, then we must search again *once*
> networking is up.
>
> Fair enough, but you can then of course not use that unit to configure
> the network.


Of course we can.  We need to cycle the network though.


> But this "if there isn't a local seed" isn't something you
> can express as a static condition, hence my thought that it might be
> better if c-i calls s-n-wait-online if and only if it's necessary. But
> YMMV.
>

Right, though it's not clear to me that we can express this in unit terms.
It may have to be done internally;  That is, it's possible that in
cloud-init "net"
mode we need to block ourselves, until networking is-up.


>
>
> > This works just fine with 'networking.service'
>
> This did/does not really work "fine" IMHO -- all of our cloud images
> hang for a long time at boot unless you give them a local data source or
> disable cloud-init.


This is by-design.  cloud-init is *interposing* itself on purpose.


> It also imposes the restriction that you must be
>
online during boot, which is fine for a cloud environment, but rather
> unfriendly for other scenarios.
>

No, you need provide a datasource, or indicate (via boot params) that
you're not
interested in cloud-init running.

It's certainly true that if someone just qemu-system-x86 -hda cloud.img
that it's going
to hang.  But folks are explicitly booting a *cloud* image without a cloud.

We handle this fine with uvt-kvm which provides a nocloud-net seed when
booting.


> > due to the "atomic" nature of ifup where once the oneshot service
> runs, we can assume that networking is up. However, networkd runs and
> asynchronously brings up networking; which is fine but we now no longer
> have a clear checkpoint at which cloud-init can run with networking up
> but before.
>
> Again -- s-n-wait-online.service is exactly the networkd counterpart of
> networking.service for ifupdown, that gives you the "network is fully
> configured" synchronization point. The issue is not that it doesn't
> exist, but that I think that it's not a good thing to depend on either
> one.
>

It is, but it's a separate unit "networking" == "networkd" +
"networkd-wait-online"
However, netplan generator only emits the "systemd-networkd" target wants,
so
if we use After=systemd-networkd-wait-online; that's never run since
nothing wants it.
If we add it explicitly, then it runs even when networkd doesn't


>
> > we really want something like
> >  After=networking|networkd-wait-online
> > which handles determining if networkd was supposed to run or not
>
> That already exists, it's network-online.target -- whatever "implements"
> it (ifupdown, networkd, NM) will hook itself into this target. Nothing
> more, nothing less, so if cloud-init just wants to wait until it's
> online, then just make it Requires/After=network-online.target instead
> of Before= it. (But again -- this is a very strong dependency which is
> very inconvenient anywhere but cloud environments with essentially one
> virtual ethernet card).
>

It may be that network-online.target is the right place.  Scott had some
reason
for not using that explicitly before;  I expect some details from him.

It's no more inconvenient than cloud-init has ever been for users not
providing
a data-source, or booting with cloud-init disabled.


>
> BTW, I'm not sure if it came across -- if you play around with this,
> please drop systemd-networkd.service's After=dbus.service; that will get
> rid of the worst dependency cycles, and it's something which we can do
> in Xenial rather easily (not so easy for devel, that's the part we need
> to discuss with upstream or decide if we care enough about this feature,
> but eventually I figure we want to get rid of it either way).
>

I did play with it, but the networkd in xenial blocks for some non-trivial
amount of time (10s of seconds)
if dbus.service is not up.


>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1636912
>
> Title:
>   systemd-networkd runs too late for cloud-init.service (net)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1636912/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  New
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-27 Thread Ryan Harper 
On Thu, Oct 27, 2016 at 3:57 PM, Martin Pitt 
wrote:

> > cloud-init expects networking to be up, like 'networking.service'
> before it runs..
>
>
> It seems to me that this might have to be split into two parts then -- one
> that can provide network config which runs early and does not require
> networking, and one that can use the network to configure other bits?
>

We already do this.  cloud-init-local.service runs before networking; it
examines for *local* non-network seeds (like nocloud-net, or a config-drive
)
If that's present, it's sourced and used.

However, if there isn't a local seed, then we must search again *once*
networking is up.

This works just fine with 'networking.service' due to the "atomic" nature
of ifup where once
the oneshot service runs, we can assume that networking is up.

However, networkd runs and asynchronously brings up networking; which is
fine but we now
no longer have a clear checkpoint at which cloud-init can run with
networking up but before
we're at the full 'network-online.target'

I'm not sure how to close the subtle distinction between 'networking' and
'systemd-networkd'
but it's clearly different with no obvious way to make them equivalent.

>
> > So why shouldn't we use networkd-wait-online ?
>
> You can use the program. I said that it might not be the best idea to
> use After=s-n-wait-online.service, as that would block the entire cloud-
> init.service and with it the entire boot (as cloud-init.service has very
> strong dependencies) if there is no network available.
>

It actually works quite well, except the netplan generator only creates a
wants for systemd-networkd, so nothing *wants* the netword-wait-online
unless we add it; this is problematic for cloud-init on a system which
doesn't
have a netplan config (where networkd isn't going to run).


>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1636912
>
> Title:
>   systemd-networkd runs too late for cloud-init.service (net)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1636912/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  Unknown
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
└─basic.target @11.403s
  └─sockets.target @11.401s
└─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
  └─network-pre.target @9.295s

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-27 Thread Ryan Harper 
On Thu, Oct 27, 2016 at 3:57 PM, Martin Pitt 
wrote:

> > cloud-init expects networking to be up, like 'networking.service'
> before it runs..
>
>
> I think it needs to make up its mind -- why does it want to run
> Before=network-online.target then? I thought the idea was that cloud-init
> is able to *provide* a network configuration (through the YAML). It seems
> to me that this might have to be split into two parts then -- one that can
> provide network config which runs early and does not require networking,
> and one that can use the network to configure other bits?
>

we have to do both.

In the case that we have a local config which provides networking
configuration, we can emit it, and then we want to "bring it up"

In other cases, we may need to bring up fallback networking (ie, dhcp on
eth0) to look for a metadata service on the network
which also may provide network configuration (which we writeout) and then
want to "bring it up"


>
> > So why shouldn't we use networkd-wait-online ?
>
> You can use the program. I said that it might not be the best idea to
> use After=s-n-wait-online.service, as that would block the entire cloud-
> init.service and with it the entire boot (as cloud-init.service has very
> strong dependencies) if there is no network available.
>

cloud-init.service *requires* network; it's designed to block until then
otherwise
you just timeout trying to reach network endpoints like Openstack metadata
service.

The "hang" in UbuntuCore was primarily related to *not* providing a nocloud
seed
in the image;  when it was booted without a seed and not disabled, it goes
an *looks* for a seed, first local, then over the net.


>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1636912
>
> Title:
>   systemd-networkd runs too late for cloud-init.service (net)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1636912/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  Unknown
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
└─basic.target @11.403s
  └─sockets.target @11.401s
└─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
└─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
└─run-cgmanager-fs.mount @12.687s

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-27 Thread Ryan Harper 
On Thu, Oct 27, 2016 at 2:50 PM, Martin Pitt 
wrote:

> > oddly though when using the ifupdown 'networking.service'; we don't
> need to use that target.
>
> Yes, that's a Type=oneshot, as it just calls "ifup -a". So that's more
> or less equivalent to s-n-wait-online --timeout=30 or After=s-n-wait-
> online.service. But the latter would block the entire boot process for
> that long if there is no network (and this *did* hit us in snappy
> already, like bug 1431836) -- my gut feeling is that this can be handled
> more gracefully/asynchronously in code.
>

Where though?  cloud-init expects networking to be up, like
'networking.service'
before it runs..  So why shouldn't we use networkd-wait-online ?

Additionally, cloud-init needs to wait for networking to be up, whether the
system
is using ifupdown/networking.service or netplan/networkd ... Adding After=
for both of these
appears to be problematic;  we really want something like

After=networking|networkd-wait-online

which handles determining if networkd was supposed to run or not

Maybe a Conditional After would be nice here; we could see if networkd was
expected to start

It's possible that this isn't an issue outside of Ubuntu Core 16.

For Xenial cloud-images, we don't yet have networkd/resolved/ and netplan
to replace ifupdown setup
For Y+ cloud-images, we can moved to that if we want since all of the parts
are there too

For UC16 on Xenial, it *does* have networkd/netplan and expects to use that
by default; however it
currently comes in with a dep on ifupdown which could be dropped if
cloud-init has enough support
for network yaml v2/netplan for fallback networking (though the UC16 image
has a built-in network
config like the older cloud-images did).

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  Unknown
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
└─basic.target @11.403s
  └─sockets.target @11.401s
└─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
└─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
└─run-cgmanager-fs.mount @12.687s
  └─local-fs-pre.target @1.393s
└─systemd-tmpfiles-setup-dev.service @1.116s +195ms
  └─kmod-static-nodes.service @887ms +193ms
└─system.slice @783ms

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-27 Thread Steve Langasek 
On Thu, Oct 27, 2016 at 01:43:05PM -, Ryan Harper wrote:

> without resolved in 16.04 though the change isn't strictly needed but to
> keep the backported branch closer to trunk, I would expect those to come
> in.

Except it's not correct that we don't have resolved in 16.04.  We're not
using resolved in the *desktop or server* in 16.04.  Ubuntu Core 16 has
explicitly seeded libnss-resolve in their image, you can see it configured
in /etc/nsswitch.conf; for some reason the symlink that libnss-resolve sets
up for /etc/systemd/system/multi-user.target.wants/systemd-resolved.service
→ /lib/systemd/system/systemd-resolved.service is absent, but this may be a
bug related to the bind mounts over /etc/systemd and maybe shouldn't be
relied on.  I certainly don't know what to expect from the current resolved
setup in Ubuntu Core 16, combined with the cloud-init systemd changes from
zesty.

(Martin, I don't suppose the snappy team talked to you before seeding
libnss-resolve, which is a universe package in xenial, in their images...?)

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  Unknown
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
└─basic.target @11.403s
  └─sockets.target @11.401s
└─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
└─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
└─run-cgmanager-fs.mount @12.687s
  └─local-fs-pre.target @1.393s
└─systemd-tmpfiles-setup-dev.service @1.116s +195ms
  └─kmod-static-nodes.service @887ms +193ms
└─system.slice @783ms
  └─-.slice @721ms

  
  cloud-init would need networkd to run at or before 'networking.service' so it 
can raise networking to then find and use network-based datasources.

  # grep systemd /usr/share/snappy/dpkg.list 
  ii  libnss-resolve:amd64  229-4ubuntu11   
 amd64nss module to resolve names via systemd-resolved
  ii  libpam-systemd:amd64  229-4ubuntu11   
 amd64system and service manager - PAM module
  ii  libsystemd0:amd64 229-4ubuntu11   
 amd64systemd utility library
  ii  systemd   229-4ubuntu11

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-27 Thread Ryan Harper 
On Thu, Oct 27, 2016 at 8:43 AM, Ryan Harper 
wrote:

>
>
> On Thu, Oct 27, 2016 at 5:59 AM, Martin Pitt 
> wrote:
>
>> So did I understand this right:
>>
>>  * In current xenial, cloud-init runs in late boot, so there is no
>> principal ordering problem between cloud-init and networkd, other than
>> that cloud-init.service should declare After=systemd-networkd.service
>> similar to what it does for ifupdown (After=networking.service).
>>
>>
I'm playing with this now, and we've got some more ordering to do.

I've used After and Requires; but these are focused on when the units starts
rather than when we can expect networking to be up.

networkd runs but it takes a few seconds to get DHCP responses and have
the interfaces come up.  What I'm seeing is that systemd runs networkd, this
then allows cloud-init.service to run; which it then checks on the network
interfaces
and finds that eth0 isn't up yet, several seconds later eth0 does come up
but not
before cloud-init.service runs.

There is a network-online.target, which I think we'd want to say
cloud-init.service runs
After that; oddly though when using the ifupdown 'networking.service'; we
don't need
to use that target. and cloud-init.service explicitly runs
Before=network-online.target

That seems wrong since cloud-init.service may scan for network metadata
services.
If I remove that line, I'll see how that affects both the ifupdown and
networkd path.

Ryan

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  Unknown
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
└─basic.target @11.403s
  └─sockets.target @11.401s
└─dbus.socket @11.398s
  └─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
  └─network-pre.target @9.295s
└─cloud-init-local.service @3.822s +5.469s
  └─local-fs.target @3.813s
└─run-cgmanager-fs.mount @12.687s
  └─local-fs-pre.target @1.393s
└─systemd-tmpfiles-setup-dev.service @1.116s +195ms
  └─kmod-static-nodes.service @887ms +193ms
└─system.slice @783ms
  └─-.slice @721ms

  
  cloud-init would need networkd to run at or before 'networking.service' so it 
can raise networking to then find and use network-based datasources.

  # grep systemd /usr/share/snappy/dpkg.list 
  ii  libnss-resolve:amd64  229-4ubuntu11

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-27 Thread Ryan Harper 
On Thu, Oct 27, 2016 at 5:59 AM, Martin Pitt 
wrote:

> So did I understand this right:
>
>  * In current xenial, cloud-init runs in late boot, so there is no
> principal ordering problem between cloud-init and networkd, other than
> that cloud-init.service should declare After=systemd-networkd.service
> similar to what it does for ifupdown (After=networking.service).
>
>  * Thus this would not be a blocker for Ubuntu Core 16 right now, and
> merely adding that After= should suffice for the moment.
>
>  * This *is* an issue for 16.10/zesty as cloud-init.service runs in
> early boot and dbus/networkd can't.
>

I suppose we could cherry pick the upstream snap create-user feature and
other
fixes but I expect the SRU of cloud-init to Xenial to include the change in
ordering
to enable running cloud-init earlier in boot;  without resolved in 16.04
though the
change isn't strictly needed but to keep the backported branch closer to
trunk, I
would expect those to come in.

However, it does sound like it's reasonable to just add the
After=systemd-networkd.service
as you say; I can test this out now and confirm if that's sufficient to get
a picture of what
this should look like.


>
>  * This *will become* an issue for 16.04 as these cloud-init changes are
> meant to be backported soon. Will that happen for the Ubuntu Core 16 GA
> in a few days already? I. e. is this something which we need to crowbar
> in ASAP, or do we have some time to figure out a proper solution how to
> start networkd first, and dbus later on?
>

The official Ubuntu Core 16 GA images won't have cloud-init enabled by
default.
I'm currently working on a separate build of UC16 *with* cloud-init
enabled, focusing
on primary public cloud support.  In this build, I'll include at least
snapshot of cloud-init
trunk as of a few days ago to include the snap create-user support needed
for function.

I can add a task against cloud-init for this bug and get a MR with the
change to the
cloud-init.service file after testing that the changes achieves the goal.


>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1636912
>
> Title:
>   systemd-networkd runs too late for cloud-init.service (net)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+
> bug/1636912/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in systemd:
  Unknown
Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged
Status in cloud-init source package in Xenial:
  New
Status in systemd source package in Xenial:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s

Re: [Touch-packages] [Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

2016-10-26 Thread Ryan Harper 
On Wed, Oct 26, 2016 at 6:41 PM, Steve Langasek <
steve.langa...@canonical.com> wrote:

> Note that the systemd dependencies shown for cloud-init in xenial (on
> which Ubuntu Core 16 is based) don't match those listed in the bug
> description.  Instead, xenial currently has:
>
> After=cloud-init-local.service networking.service
> Before=network-online.target sshd.service sshd-keygen.service
> systemd-user-sessions.service
> Requires=networking.service
> Wants=local-fs.target cloud-init-local.service sshd.service
> sshd-keygen.service
>
> That's certainly an easier target for fixing the ordering on.
>

We're preparing an SRU for Xenial, but the issue is related to
systemd-resolved which is NOT in Xenial, so the
failing behavior of a 120 second timeout for resolving hosts when dbus
isn't up yet doesn't materialize on Xenial, only yakkety.

 https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1629797


>
> I'm surprised that cloud-init is being ordered before dbus in 16.10+.
>  This appears to have been done in response to bug #1629797.  This change
> to ordering on account of nss-resolve does not give me the warm fuzzies.
> There's feedback in the end of that bug that cloud-init should *not* be
> using Before=basic.target / Before=dbus.socket, but instead use
> Before=sysinit.target.  That seems like a change we should be making.  But
> I also didn't understand that we would be using nss-resolve - I missed a
> nuance in  June/039406.html>, I thought that once resolved supported DNS, we would
> use that /instead of/ the NSS module.  Is there really a benefit to using
> both?
>

systemd-resolved doesn't runs soon enough where cloud-init needs to resolve
metadata service end points, like in GCE.

I don't know enough to say if resolved and NSS are 100% swappable; but at a
min, resolved would need to be able to run as early as NSS runs to allow
cloud-init to resolve metadata service URLS very early in boot.


>
> Regardless, none of that seems to account for the specific problem
> reported here, on Ubuntu Core 16; because Ubuntu Core 16 does contain
> libnss-resolve, but does *not* contain the cloud-init with the
> DefaultDependencies=no.
>

We're specifically introducing a newer cloud-init which adds support for
snap create-user support which also includes the change in Unit
dependencies from upstream cloud-init.
Those changes to unit dependencies will also be SRU to Xenial.


>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1636912
>
> Title:
>   systemd-networkd runs too late for cloud-init.service (net)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+
> bug/1636912/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target 
systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root@snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" 
character.
  The time the unit takes to start is printed after the "+" character.

  root@snap-test7:~#