Re: [systemd-devel] Setting up a VPN daemon as a Portable Service

[Luca Boccassi]

On Mon, 10 Oct 2022 at 04:00, Duncan Gibson  wrote:
>
> Final update, hopefully. Here's a gist with a script, service unit, and 
> readme.

Again, that is not safe and it will fail at some point as it is open
to race conditions. You have to use ExtensionImages= instead.

> On Sun, Oct 9, 2022 at 11:10 AM Duncan Gibson  wrote:
>>
>> After doing some more looking, it seems like the /etc folder is overlaid 
>> with /var/lib/overlays/etc/upper, meaning that changes to /etc/ get saved in 
>> the overlay, which should survive updates. I've added the service definition 
>> there and the binaries to my home directory (and updated the service 
>> definition to point there), and it seems to just work. I might look into 
>> putting the binaries in a system extension, and update the service 
>> definition to require the extension be running, but it seems to work as-is. 
>> Time to wait for the next system update and see if it breaks.
>>
>> On Sat, Oct 8, 2022 at 2:02 PM Luca Boccassi  wrote:
>>>
>>> On Sat, 8 Oct 2022 at 18:51, Duncan Gibson  wrote:
>>> >
>>> > Hm. Actually, no, I don't think it will. Services installed the normal 
>>> > way won't survive the A/B update system.
>>>
>>> I don't know what that is and how it works.
>>>
>>> > On Sat, Oct 8, 2022 at 12:23 PM Duncan Gibson  
>>> > wrote:
>>> >>
>>> >> Oh, now that's a new way of doing it. I'll definitely give that a shot. 
>>> >> That sounds like it has the best chance of working.
>>> >>
>>> >> On Sat, Oct 8, 2022 at 12:20 PM Luca Boccassi  wrote:
>>> >>>
>>> >>> On Sat, 2022-10-08 at 11:13 -0400, Duncan Gibson wrote:
>>> >>> > The problem wasn't mounting the system extension automatically. That
>>> >>> > worked
>>> >>> > just fine. It was that systemd would try to start the service before
>>> >>> > the
>>> >>> > system extension mounted, which would fail, for obvious reasons. This
>>> >>> > weekend I think I'm going to try the BindReadOnlyPaths option and see
>>> >>> > if I
>>> >>> > can get that to work.
>>> >>>
>>> >>> Don't do that. system-wide extensions are not supposed to add units,
>>> >>> and it will not work. Portable services are for distributors - for
>>> >>> locally built extensions, you can simply use a normal service with
>>> >>> ExtensionImages= that points to your extension, and it will be
>>> >>> overlayed with the rootfs.
>>> >>>
>>> >>> > On Fri, Oct 7, 2022 at 3:35 PM David Anderson 
>>> >>> > wrote:
>>> >>> >
>>> >>> > > Yeah, so far we (tailscale) haven't found a good way to run on the
>>> >>> > > Steam
>>> >>> > > Deck at bootup, and also survive the A/B OS updates. Systemd system
>>> >>> > > extensions _can_ be activated during bootup, if you place the
>>> >>> > > extension in
>>> >>> > > one of the well-known locations (/var/lib/extensions would be the
>>> >>> > > one to
>>> >>> > > use on Deck, as iirc it survives A/B upgrades), and the systemd-
>>> >>> > > sysext
>>> >>> > > service is enabled.
>>> >>> > >
>>> >>> > > I would check if systemd-sysext.service is enabled on the deck, and
>>> >>> > > if
>>> >>> > > not, file a request with Valve to enable that service in a future
>>> >>> > > update.
>>> >>> > > You should present it as enabling further customization of their
>>> >>> > > platform.
>>> >>> > >
>>> >>> > > Another possible reason that sysexts aren't working for you, is
>>> >>> > > that the
>>> >>> > > Deck's /etc/os-release doesn't define a SYSEXT_LEVEL, and the
>>> >>> > > VERSION_ID
>>> >>> > > changes with every OS update. Because of this, the system extension
>>> >>> > > will
>>> >>> > > refuse to activate after every update (either SYSEXT_LEVEL or
>>> >>> > > VERSION_ID
>>> >>> > > must match exactly), until you rebuild a new image with the right
>>> >>> > > OS
>>> >>> > > metadata. Asking Valve to set SYSEXT_LEVEL to a stable value would
>>> >>> > > make it
>>> >>> > > even easier to provide Deck OS extensions reliably :)
>>> >>> > >
>>> >>> > > - Dave
>>> >>> > >
>>> >>> > > On Thu, Oct 6, 2022, at 12:08, Arian van Putten wrote:
>>> >>> > >
>>> >>> > > Afaik Portable services run in an isolated root and dont have
>>> >>> > > access to
>>> >>> > > the hosts rootfs.  You'd have go include iptables and all its
>>> >>> > > dependencies
>>> >>> > > in the portable services directory. If you don't want to do that
>>> >>> > > you'd have
>>> >>> > > to use BindReadOnlyPaths= to give the service access to the
>>> >>> > > required host
>>> >>> > > paths or you'd have to use a system extension.
>>> >>> > >
>>> >>> > > That's probably why they advice running as a system extension.
>>> >>> > >
>>> >>> > > I think there are mechanisms for setting up system extensions on
>>> >>> > > startup
>>> >>> > > but I'm not familiar enough with the details. Maybe someone else in
>>> >>> > > the
>>> >>> > > list knows.
>>> >>> > >
>>> >>> > >
>>> >>> > >
>>> >>> > >
>>> >>> > > On Thu, 6 Oct 2022, 20:21 Duncan Gibson, 
>>> >>> > > wrote:
>>> >>> > >
>>> >>> > > Hi, everyone.
>>> >>> > >
>>> >>> > > The high-level overview: I'm trying to 

Re: [systemd-devel] Setting up a VPN daemon as a Portable Service

[Duncan Gibson]

Final update, hopefully. Here's a gist with a script, service unit, and
readme.


On Sun, Oct 9, 2022 at 11:10 AM Duncan Gibson 
wrote:

> After doing some more looking, it seems like the /etc folder is overlaid
> with /var/lib/overlays/etc/upper, meaning that changes to /etc/ get saved
> in the overlay, which should survive updates. I've added the service
> definition there and the binaries to my home directory (and updated the
> service definition to point there), and it seems to just work. I might look
> into putting the binaries in a system extension, and update the service
> definition to require the extension be running, but it seems to work as-is.
> Time to wait for the next system update and see if it breaks.
>
> On Sat, Oct 8, 2022 at 2:02 PM Luca Boccassi  wrote:
>
>> On Sat, 8 Oct 2022 at 18:51, Duncan Gibson 
>> wrote:
>> >
>> > Hm. Actually, no, I don't think it will. Services installed the normal
>> way won't survive the A/B update system.
>>
>> I don't know what that is and how it works.
>>
>> > On Sat, Oct 8, 2022 at 12:23 PM Duncan Gibson 
>> wrote:
>> >>
>> >> Oh, now that's a new way of doing it. I'll definitely give that a
>> shot. That sounds like it has the best chance of working.
>> >>
>> >> On Sat, Oct 8, 2022 at 12:20 PM Luca Boccassi 
>> wrote:
>> >>>
>> >>> On Sat, 2022-10-08 at 11:13 -0400, Duncan Gibson wrote:
>> >>> > The problem wasn't mounting the system extension automatically. That
>> >>> > worked
>> >>> > just fine. It was that systemd would try to start the service before
>> >>> > the
>> >>> > system extension mounted, which would fail, for obvious reasons.
>> This
>> >>> > weekend I think I'm going to try the BindReadOnlyPaths option and
>> see
>> >>> > if I
>> >>> > can get that to work.
>> >>>
>> >>> Don't do that. system-wide extensions are not supposed to add units,
>> >>> and it will not work. Portable services are for distributors - for
>> >>> locally built extensions, you can simply use a normal service with
>> >>> ExtensionImages= that points to your extension, and it will be
>> >>> overlayed with the rootfs.
>> >>>
>> >>> > On Fri, Oct 7, 2022 at 3:35 PM David Anderson 
>> >>> > wrote:
>> >>> >
>> >>> > > Yeah, so far we (tailscale) haven't found a good way to run on the
>> >>> > > Steam
>> >>> > > Deck at bootup, and also survive the A/B OS updates. Systemd
>> system
>> >>> > > extensions _can_ be activated during bootup, if you place the
>> >>> > > extension in
>> >>> > > one of the well-known locations (/var/lib/extensions would be the
>> >>> > > one to
>> >>> > > use on Deck, as iirc it survives A/B upgrades), and the systemd-
>> >>> > > sysext
>> >>> > > service is enabled.
>> >>> > >
>> >>> > > I would check if systemd-sysext.service is enabled on the deck,
>> and
>> >>> > > if
>> >>> > > not, file a request with Valve to enable that service in a future
>> >>> > > update.
>> >>> > > You should present it as enabling further customization of their
>> >>> > > platform.
>> >>> > >
>> >>> > > Another possible reason that sysexts aren't working for you, is
>> >>> > > that the
>> >>> > > Deck's /etc/os-release doesn't define a SYSEXT_LEVEL, and the
>> >>> > > VERSION_ID
>> >>> > > changes with every OS update. Because of this, the system
>> extension
>> >>> > > will
>> >>> > > refuse to activate after every update (either SYSEXT_LEVEL or
>> >>> > > VERSION_ID
>> >>> > > must match exactly), until you rebuild a new image with the right
>> >>> > > OS
>> >>> > > metadata. Asking Valve to set SYSEXT_LEVEL to a stable value would
>> >>> > > make it
>> >>> > > even easier to provide Deck OS extensions reliably :)
>> >>> > >
>> >>> > > - Dave
>> >>> > >
>> >>> > > On Thu, Oct 6, 2022, at 12:08, Arian van Putten wrote:
>> >>> > >
>> >>> > > Afaik Portable services run in an isolated root and dont have
>> >>> > > access to
>> >>> > > the hosts rootfs.  You'd have go include iptables and all its
>> >>> > > dependencies
>> >>> > > in the portable services directory. If you don't want to do that
>> >>> > > you'd have
>> >>> > > to use BindReadOnlyPaths= to give the service access to the
>> >>> > > required host
>> >>> > > paths or you'd have to use a system extension.
>> >>> > >
>> >>> > > That's probably why they advice running as a system extension.
>> >>> > >
>> >>> > > I think there are mechanisms for setting up system extensions on
>> >>> > > startup
>> >>> > > but I'm not familiar enough with the details. Maybe someone else
>> in
>> >>> > > the
>> >>> > > list knows.
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > > On Thu, 6 Oct 2022, 20:21 Duncan Gibson, 
>> >>> > > wrote:
>> >>> > >
>> >>> > > Hi, everyone.
>> >>> > >
>> >>> > > The high-level overview: I'm trying to install Tailscale as a
>> >>> > > portable
>> >>> > > service on my Steam Deck.
>> >>> > >
>> >>> > > Tailscale is a point-to-point VPN service, essentially a wrapper
>> >>> > > around
>> >>> > > Wireguard that helps with 

Re: [systemd-devel] Setting up a VPN daemon as a Portable Service

[Duncan Gibson]

After doing some more looking, it seems like the /etc folder is overlaid
with /var/lib/overlays/etc/upper, meaning that changes to /etc/ get saved
in the overlay, which should survive updates. I've added the service
definition there and the binaries to my home directory (and updated the
service definition to point there), and it seems to just work. I might look
into putting the binaries in a system extension, and update the service
definition to require the extension be running, but it seems to work as-is.
Time to wait for the next system update and see if it breaks.

On Sat, Oct 8, 2022 at 2:02 PM Luca Boccassi  wrote:

> On Sat, 8 Oct 2022 at 18:51, Duncan Gibson  wrote:
> >
> > Hm. Actually, no, I don't think it will. Services installed the normal
> way won't survive the A/B update system.
>
> I don't know what that is and how it works.
>
> > On Sat, Oct 8, 2022 at 12:23 PM Duncan Gibson 
> wrote:
> >>
> >> Oh, now that's a new way of doing it. I'll definitely give that a shot.
> That sounds like it has the best chance of working.
> >>
> >> On Sat, Oct 8, 2022 at 12:20 PM Luca Boccassi  wrote:
> >>>
> >>> On Sat, 2022-10-08 at 11:13 -0400, Duncan Gibson wrote:
> >>> > The problem wasn't mounting the system extension automatically. That
> >>> > worked
> >>> > just fine. It was that systemd would try to start the service before
> >>> > the
> >>> > system extension mounted, which would fail, for obvious reasons. This
> >>> > weekend I think I'm going to try the BindReadOnlyPaths option and see
> >>> > if I
> >>> > can get that to work.
> >>>
> >>> Don't do that. system-wide extensions are not supposed to add units,
> >>> and it will not work. Portable services are for distributors - for
> >>> locally built extensions, you can simply use a normal service with
> >>> ExtensionImages= that points to your extension, and it will be
> >>> overlayed with the rootfs.
> >>>
> >>> > On Fri, Oct 7, 2022 at 3:35 PM David Anderson 
> >>> > wrote:
> >>> >
> >>> > > Yeah, so far we (tailscale) haven't found a good way to run on the
> >>> > > Steam
> >>> > > Deck at bootup, and also survive the A/B OS updates. Systemd system
> >>> > > extensions _can_ be activated during bootup, if you place the
> >>> > > extension in
> >>> > > one of the well-known locations (/var/lib/extensions would be the
> >>> > > one to
> >>> > > use on Deck, as iirc it survives A/B upgrades), and the systemd-
> >>> > > sysext
> >>> > > service is enabled.
> >>> > >
> >>> > > I would check if systemd-sysext.service is enabled on the deck, and
> >>> > > if
> >>> > > not, file a request with Valve to enable that service in a future
> >>> > > update.
> >>> > > You should present it as enabling further customization of their
> >>> > > platform.
> >>> > >
> >>> > > Another possible reason that sysexts aren't working for you, is
> >>> > > that the
> >>> > > Deck's /etc/os-release doesn't define a SYSEXT_LEVEL, and the
> >>> > > VERSION_ID
> >>> > > changes with every OS update. Because of this, the system extension
> >>> > > will
> >>> > > refuse to activate after every update (either SYSEXT_LEVEL or
> >>> > > VERSION_ID
> >>> > > must match exactly), until you rebuild a new image with the right
> >>> > > OS
> >>> > > metadata. Asking Valve to set SYSEXT_LEVEL to a stable value would
> >>> > > make it
> >>> > > even easier to provide Deck OS extensions reliably :)
> >>> > >
> >>> > > - Dave
> >>> > >
> >>> > > On Thu, Oct 6, 2022, at 12:08, Arian van Putten wrote:
> >>> > >
> >>> > > Afaik Portable services run in an isolated root and dont have
> >>> > > access to
> >>> > > the hosts rootfs.  You'd have go include iptables and all its
> >>> > > dependencies
> >>> > > in the portable services directory. If you don't want to do that
> >>> > > you'd have
> >>> > > to use BindReadOnlyPaths= to give the service access to the
> >>> > > required host
> >>> > > paths or you'd have to use a system extension.
> >>> > >
> >>> > > That's probably why they advice running as a system extension.
> >>> > >
> >>> > > I think there are mechanisms for setting up system extensions on
> >>> > > startup
> >>> > > but I'm not familiar enough with the details. Maybe someone else in
> >>> > > the
> >>> > > list knows.
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Thu, 6 Oct 2022, 20:21 Duncan Gibson, 
> >>> > > wrote:
> >>> > >
> >>> > > Hi, everyone.
> >>> > >
> >>> > > The high-level overview: I'm trying to install Tailscale as a
> >>> > > portable
> >>> > > service on my Steam Deck.
> >>> > >
> >>> > > Tailscale is a point-to-point VPN service, essentially a wrapper
> >>> > > around
> >>> > > Wireguard that helps with network setup and management. The Steam
> >>> > > Deck is
> >>> > > Valve's handheld PC running SteamOS 3, which is derived from Arch.
> >>> > > It uses
> >>> > > an A/B partition system for system files, meaning you can't install
> >>> > > a
> >>> > > service the normal way.
> >>> > >
> >>> > > There *is* a guide to do this, posted on their own blog, but 

Re: [systemd-devel] Setting up a VPN daemon as a Portable Service

[Duncan Gibson]

Oh, now that's a new way of doing it. I'll definitely give that a shot.
That sounds like it has the best chance of working.

On Sat, Oct 8, 2022 at 12:20 PM Luca Boccassi  wrote:

> On Sat, 2022-10-08 at 11:13 -0400, Duncan Gibson wrote:
> > The problem wasn't mounting the system extension automatically. That
> > worked
> > just fine. It was that systemd would try to start the service before
> > the
> > system extension mounted, which would fail, for obvious reasons. This
> > weekend I think I'm going to try the BindReadOnlyPaths option and see
> > if I
> > can get that to work.
>
> Don't do that. system-wide extensions are not supposed to add units,
> and it will not work. Portable services are for distributors - for
> locally built extensions, you can simply use a normal service with
> ExtensionImages= that points to your extension, and it will be
> overlayed with the rootfs.
>
> > On Fri, Oct 7, 2022 at 3:35 PM David Anderson 
> > wrote:
> >
> > > Yeah, so far we (tailscale) haven't found a good way to run on the
> > > Steam
> > > Deck at bootup, and also survive the A/B OS updates. Systemd system
> > > extensions _can_ be activated during bootup, if you place the
> > > extension in
> > > one of the well-known locations (/var/lib/extensions would be the
> > > one to
> > > use on Deck, as iirc it survives A/B upgrades), and the systemd-
> > > sysext
> > > service is enabled.
> > >
> > > I would check if systemd-sysext.service is enabled on the deck, and
> > > if
> > > not, file a request with Valve to enable that service in a future
> > > update.
> > > You should present it as enabling further customization of their
> > > platform.
> > >
> > > Another possible reason that sysexts aren't working for you, is
> > > that the
> > > Deck's /etc/os-release doesn't define a SYSEXT_LEVEL, and the
> > > VERSION_ID
> > > changes with every OS update. Because of this, the system extension
> > > will
> > > refuse to activate after every update (either SYSEXT_LEVEL or
> > > VERSION_ID
> > > must match exactly), until you rebuild a new image with the right
> > > OS
> > > metadata. Asking Valve to set SYSEXT_LEVEL to a stable value would
> > > make it
> > > even easier to provide Deck OS extensions reliably :)
> > >
> > > - Dave
> > >
> > > On Thu, Oct 6, 2022, at 12:08, Arian van Putten wrote:
> > >
> > > Afaik Portable services run in an isolated root and dont have
> > > access to
> > > the hosts rootfs.  You'd have go include iptables and all its
> > > dependencies
> > > in the portable services directory. If you don't want to do that
> > > you'd have
> > > to use BindReadOnlyPaths= to give the service access to the
> > > required host
> > > paths or you'd have to use a system extension.
> > >
> > > That's probably why they advice running as a system extension.
> > >
> > > I think there are mechanisms for setting up system extensions on
> > > startup
> > > but I'm not familiar enough with the details. Maybe someone else in
> > > the
> > > list knows.
> > >
> > >
> > >
> > >
> > > On Thu, 6 Oct 2022, 20:21 Duncan Gibson, 
> > > wrote:
> > >
> > > Hi, everyone.
> > >
> > > The high-level overview: I'm trying to install Tailscale as a
> > > portable
> > > service on my Steam Deck.
> > >
> > > Tailscale is a point-to-point VPN service, essentially a wrapper
> > > around
> > > Wireguard that helps with network setup and management. The Steam
> > > Deck is
> > > Valve's handheld PC running SteamOS 3, which is derived from Arch.
> > > It uses
> > > an A/B partition system for system files, meaning you can't install
> > > a
> > > service the normal way.
> > >
> > > There *is* a guide to do this, posted on their own blog, but it
> > > uses
> > > system extensions which aren't good for services that you want to
> > > run on
> > > startup. Indeed, following that guide puts me in a state where I
> > > have to
> > > manually start the daemon every time I reboot my Deck, even with
> > > the
> > > service enabled.
> > >
> > > Let's move on to how I've started to do this.
> > >
> > > Tailscale is available through most package managers, but they also
> > > publish static binaries with systemd unit files.
> > >
> > > This script grabs that binary, extracts it, and moves it into a
> > > portable
> > > service directory structure.
> > >
> > > # download and extract Tailscale
> > > tarball="$(curl -s 'https://pkgs.tailscale.com/stable/?mode=json' |
> > > jq -r
> > > .Tarballs.amd64)"
> > > version="$(echo ${tarball} | cut -d_ -f2)"
> > > tar_dir="$(echo ${tarball} | cut -d. -f1-3)"
> > > curl -s "https://pkgs.tailscale.com/stable/${tarball}; -o
> > > tailscale.tgz
> > > tar xzf tailscale.tgz
> > > test -d $tar_dir
> > >
> > > # Set up our target directory structure
> > > mkdir -p
> > > tailscaled/{usr/{bin,sbin,lib/systemd/system},etc,proc,sys,dev,run,
> > > /var/tmp}
> > >
> > > # Copy tailscale-distributed files to the right place
> > > cp -rf $tar_dir/tailscaled tailscaled/usr/sbin/tailscaled
> > > cp -rf 

Re: [systemd-devel] Setting up a VPN daemon as a Portable Service

[Luca Boccassi]

On Sat, 2022-10-08 at 11:13 -0400, Duncan Gibson wrote:
> The problem wasn't mounting the system extension automatically. That
> worked
> just fine. It was that systemd would try to start the service before
> the
> system extension mounted, which would fail, for obvious reasons. This
> weekend I think I'm going to try the BindReadOnlyPaths option and see
> if I
> can get that to work.

Don't do that. system-wide extensions are not supposed to add units,
and it will not work. Portable services are for distributors - for
locally built extensions, you can simply use a normal service with
ExtensionImages= that points to your extension, and it will be
overlayed with the rootfs.

> On Fri, Oct 7, 2022 at 3:35 PM David Anderson 
> wrote:
> 
> > Yeah, so far we (tailscale) haven't found a good way to run on the
> > Steam
> > Deck at bootup, and also survive the A/B OS updates. Systemd system
> > extensions _can_ be activated during bootup, if you place the
> > extension in
> > one of the well-known locations (/var/lib/extensions would be the
> > one to
> > use on Deck, as iirc it survives A/B upgrades), and the systemd-
> > sysext
> > service is enabled.
> > 
> > I would check if systemd-sysext.service is enabled on the deck, and
> > if
> > not, file a request with Valve to enable that service in a future
> > update.
> > You should present it as enabling further customization of their
> > platform.
> > 
> > Another possible reason that sysexts aren't working for you, is
> > that the
> > Deck's /etc/os-release doesn't define a SYSEXT_LEVEL, and the
> > VERSION_ID
> > changes with every OS update. Because of this, the system extension
> > will
> > refuse to activate after every update (either SYSEXT_LEVEL or
> > VERSION_ID
> > must match exactly), until you rebuild a new image with the right
> > OS
> > metadata. Asking Valve to set SYSEXT_LEVEL to a stable value would
> > make it
> > even easier to provide Deck OS extensions reliably :)
> > 
> > - Dave
> > 
> > On Thu, Oct 6, 2022, at 12:08, Arian van Putten wrote:
> > 
> > Afaik Portable services run in an isolated root and dont have
> > access to
> > the hosts rootfs.  You'd have go include iptables and all its
> > dependencies
> > in the portable services directory. If you don't want to do that
> > you'd have
> > to use BindReadOnlyPaths= to give the service access to the
> > required host
> > paths or you'd have to use a system extension.
> > 
> > That's probably why they advice running as a system extension.
> > 
> > I think there are mechanisms for setting up system extensions on
> > startup
> > but I'm not familiar enough with the details. Maybe someone else in
> > the
> > list knows.
> > 
> > 
> > 
> > 
> > On Thu, 6 Oct 2022, 20:21 Duncan Gibson, 
> > wrote:
> > 
> > Hi, everyone.
> > 
> > The high-level overview: I'm trying to install Tailscale as a
> > portable
> > service on my Steam Deck.
> > 
> > Tailscale is a point-to-point VPN service, essentially a wrapper
> > around
> > Wireguard that helps with network setup and management. The Steam
> > Deck is
> > Valve's handheld PC running SteamOS 3, which is derived from Arch.
> > It uses
> > an A/B partition system for system files, meaning you can't install
> > a
> > service the normal way.
> > 
> > There *is* a guide to do this, posted on their own blog, but it
> > uses
> > system extensions which aren't good for services that you want to
> > run on
> > startup. Indeed, following that guide puts me in a state where I
> > have to
> > manually start the daemon every time I reboot my Deck, even with
> > the
> > service enabled.
> > 
> > Let's move on to how I've started to do this.
> > 
> > Tailscale is available through most package managers, but they also
> > publish static binaries with systemd unit files.
> > 
> > This script grabs that binary, extracts it, and moves it into a
> > portable
> > service directory structure.
> > 
> > # download and extract Tailscale
> > tarball="$(curl -s 'https://pkgs.tailscale.com/stable/?mode=json' |
> > jq -r
> > .Tarballs.amd64)"
> > version="$(echo ${tarball} | cut -d_ -f2)"
> > tar_dir="$(echo ${tarball} | cut -d. -f1-3)"
> > curl -s "https://pkgs.tailscale.com/stable/${tarball}; -o
> > tailscale.tgz
> > tar xzf tailscale.tgz
> > test -d $tar_dir
> > 
> > # Set up our target directory structure
> > mkdir -p
> > tailscaled/{usr/{bin,sbin,lib/systemd/system},etc,proc,sys,dev,run,
> > /var/tmp}
> > 
> > # Copy tailscale-distributed files to the right place
> > cp -rf $tar_dir/tailscaled tailscaled/usr/sbin/tailscaled
> > cp -rf $tar_dir/systemd/tailscaled.service
> > tailscaled/usr/lib/systemd/system/tailscaled.service
> > 
> > # Write service os-release file
> > source /etc/os-release
> > cp -rf /etc/os-release tailscaled/etc/os-release
> > 
> > Not automated yet is patching the provided unit file - you need to
> > remove
> > the EnvironmentFile line and "--port $PORT $FLAGS" options, and add
> > [Exec]
> > Environment="PATH=/usr/bin"
> > 
> > Attach the portable 

Re: [systemd-devel] Setting up a VPN daemon as a Portable Service

[Duncan Gibson]

The problem wasn't mounting the system extension automatically. That worked
just fine. It was that systemd would try to start the service before the
system extension mounted, which would fail, for obvious reasons. This
weekend I think I'm going to try the BindReadOnlyPaths option and see if I
can get that to work.

On Fri, Oct 7, 2022 at 3:35 PM David Anderson  wrote:

> Yeah, so far we (tailscale) haven't found a good way to run on the Steam
> Deck at bootup, and also survive the A/B OS updates. Systemd system
> extensions _can_ be activated during bootup, if you place the extension in
> one of the well-known locations (/var/lib/extensions would be the one to
> use on Deck, as iirc it survives A/B upgrades), and the systemd-sysext
> service is enabled.
>
> I would check if systemd-sysext.service is enabled on the deck, and if
> not, file a request with Valve to enable that service in a future update.
> You should present it as enabling further customization of their platform.
>
> Another possible reason that sysexts aren't working for you, is that the
> Deck's /etc/os-release doesn't define a SYSEXT_LEVEL, and the VERSION_ID
> changes with every OS update. Because of this, the system extension will
> refuse to activate after every update (either SYSEXT_LEVEL or VERSION_ID
> must match exactly), until you rebuild a new image with the right OS
> metadata. Asking Valve to set SYSEXT_LEVEL to a stable value would make it
> even easier to provide Deck OS extensions reliably :)
>
> - Dave
>
> On Thu, Oct 6, 2022, at 12:08, Arian van Putten wrote:
>
> Afaik Portable services run in an isolated root and dont have access to
> the hosts rootfs.  You'd have go include iptables and all its dependencies
> in the portable services directory. If you don't want to do that you'd have
> to use BindReadOnlyPaths= to give the service access to the required host
> paths or you'd have to use a system extension.
>
> That's probably why they advice running as a system extension.
>
> I think there are mechanisms for setting up system extensions on startup
> but I'm not familiar enough with the details. Maybe someone else in the
> list knows.
>
>
>
>
> On Thu, 6 Oct 2022, 20:21 Duncan Gibson,  wrote:
>
> Hi, everyone.
>
> The high-level overview: I'm trying to install Tailscale as a portable
> service on my Steam Deck.
>
> Tailscale is a point-to-point VPN service, essentially a wrapper around
> Wireguard that helps with network setup and management. The Steam Deck is
> Valve's handheld PC running SteamOS 3, which is derived from Arch. It uses
> an A/B partition system for system files, meaning you can't install a
> service the normal way.
>
> There *is* a guide to do this, posted on their own blog, but it uses
> system extensions which aren't good for services that you want to run on
> startup. Indeed, following that guide puts me in a state where I have to
> manually start the daemon every time I reboot my Deck, even with the
> service enabled.
>
> Let's move on to how I've started to do this.
>
> Tailscale is available through most package managers, but they also
> publish static binaries with systemd unit files.
>
> This script grabs that binary, extracts it, and moves it into a portable
> service directory structure.
>
> # download and extract Tailscale
> tarball="$(curl -s 'https://pkgs.tailscale.com/stable/?mode=json' | jq -r
> .Tarballs.amd64)"
> version="$(echo ${tarball} | cut -d_ -f2)"
> tar_dir="$(echo ${tarball} | cut -d. -f1-3)"
> curl -s "https://pkgs.tailscale.com/stable/${tarball}; -o tailscale.tgz
> tar xzf tailscale.tgz
> test -d $tar_dir
>
> # Set up our target directory structure
> mkdir -p
> tailscaled/{usr/{bin,sbin,lib/systemd/system},etc,proc,sys,dev,run,/var/tmp}
>
> # Copy tailscale-distributed files to the right place
> cp -rf $tar_dir/tailscaled tailscaled/usr/sbin/tailscaled
> cp -rf $tar_dir/systemd/tailscaled.service
> tailscaled/usr/lib/systemd/system/tailscaled.service
>
> # Write service os-release file
> source /etc/os-release
> cp -rf /etc/os-release tailscaled/etc/os-release
>
> Not automated yet is patching the provided unit file - you need to remove
> the EnvironmentFile line and "--port $PORT $FLAGS" options, and add
> [Exec]
> Environment="PATH=/usr/bin"
>
> Attach the portable service: sudo portablectl attach ./tailscaled
> --profile=trusted
> and try starting it: sudo systemctl start tailscaled
>
> It fails, leaving this in the logs:
>
> logtail started
> Program starting: v1.30.2-t24c524c78-gc399ae6fa, Go 1.19.1-tsb13188dd36:
> []string{"/usr/sbin/tailscaled",
> "--state=/var/lib/tailscale/tailscaled.state",
> "--socket=/run/tailscale/tailscaled.sock"}
> LogID: 0f59ed267a2b19cc28aac9ee7119914000ca478234af8d56893a025ae72cc647
> logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
> wgengine.NewUserspaceEngine(tun "tailscale0") ...
> wgengine.NewUserspaceEngine(tun "tailscale0") error: creating router:
> could not get iptables version: fork/exec /usr/bin/iptables: no such file
> or 

Re: [systemd-devel] Setting up a VPN daemon as a Portable Service

[David Anderson]

Yeah, so far we (tailscale) haven't found a good way to run on the Steam Deck 
at bootup, and also survive the A/B OS updates. Systemd system extensions _can_ 
be activated during bootup, if you place the extension in one of the well-known 
locations (/var/lib/extensions would be the one to use on Deck, as iirc it 
survives A/B upgrades), and the systemd-sysext service is enabled.

I would check if systemd-sysext.service is enabled on the deck, and if not, 
file a request with Valve to enable that service in a future update. You should 
present it as enabling further customization of their platform.

Another possible reason that sysexts aren't working for you, is that the Deck's 
/etc/os-release doesn't define a SYSEXT_LEVEL, and the VERSION_ID changes with 
every OS update. Because of this, the system extension will refuse to activate 
after every update (either SYSEXT_LEVEL or VERSION_ID must match exactly), 
until you rebuild a new image with the right OS metadata. Asking Valve to set 
SYSEXT_LEVEL to a stable value would make it even easier to provide Deck OS 
extensions reliably :)

- Dave

On Thu, Oct 6, 2022, at 12:08, Arian van Putten wrote:
> Afaik Portable services run in an isolated root and dont have access to the 
> hosts rootfs.  You'd have go include iptables and all its dependencies in the 
> portable services directory. If you don't want to do that you'd have to use 
> BindReadOnlyPaths= to give the service access to the required host paths or 
> you'd have to use a system extension.
> 
> That's probably why they advice running as a system extension. 
> 
> I think there are mechanisms for setting up system extensions on startup but 
> I'm not familiar enough with the details. Maybe someone else in the list 
> knows.
> 
> 
> 
> 
> On Thu, 6 Oct 2022, 20:21 Duncan Gibson,  wrote:
>> Hi, everyone.
>> 
>> The high-level overview: I'm trying to install Tailscale 
>>  as a portable service on my Steam Deck. 
>> 
>> Tailscale is a point-to-point VPN service, essentially a wrapper around 
>> Wireguard that helps with network setup and management. The Steam Deck is 
>> Valve's handheld PC running SteamOS 3, which is derived from Arch. It uses 
>> an A/B partition system for system files, meaning you can't install a 
>> service the normal way.
>> 
>> There *is* a guide to do this , 
>> posted on their own blog, but it uses system extensions which aren't good 
>> for services that you want to run on startup. Indeed, following that guide 
>> puts me in a state where I have to manually start the daemon every time I 
>> reboot my Deck, even with the service enabled.
>> 
>> Let's move on to how I've started to do this.
>> 
>> Tailscale is available through most package managers, but they also publish 
>> static binaries with systemd unit files 
>> . 
>> 
>> This script grabs that binary, extracts it, and moves it into a portable 
>> service directory structure.
>> 
>> # download and extract Tailscale
>> tarball="$(curl -s 'https://pkgs.tailscale.com/stable/?mode=json' | jq -r 
>> .Tarballs.amd64)"
>> version="$(echo ${tarball} | cut -d_ -f2)"
>> tar_dir="$(echo ${tarball} | cut -d. -f1-3)"
>> curl -s "https://pkgs.tailscale.com/stable/${tarball}; -o tailscale.tgz
>> tar xzf tailscale.tgz
>> test -d $tar_dir
>> 
>> # Set up our target directory structure
>> mkdir -p 
>> tailscaled/{usr/{bin,sbin,lib/systemd/system},etc,proc,sys,dev,run,/var/tmp}
>> 
>> # Copy tailscale-distributed files to the right place
>> cp -rf $tar_dir/tailscaled tailscaled/usr/sbin/tailscaled
>> cp -rf $tar_dir/systemd/tailscaled.service 
>> tailscaled/usr/lib/systemd/system/tailscaled.service
>> 
>> # Write service os-release file
>> source /etc/os-release
>> cp -rf /etc/os-release tailscaled/etc/os-release
>> 
>> Not automated yet is patching the provided unit file - you need to remove 
>> the EnvironmentFile line and "--port $PORT $FLAGS" options, and add 
>> [Exec]
>> Environment="PATH=/usr/bin"
>> 
>> Attach the portable service: sudo portablectl attach ./tailscaled 
>> --profile=trusted
>> and try starting it: sudo systemctl start tailscaled
>> 
>> It fails, leaving this in the logs:
>> 
>> `logtail started `
>> `Program starting: v1.30.2-t24c524c78-gc399ae6fa, Go 1.19.1-tsb13188dd36: 
>> []string{"/usr/sbin/tailscaled", 
>> "--state=/var/lib/tailscale/tailscaled.state", 
>> "--socket=/run/tailscale/tailscaled.sock"} `
>> `LogID: 0f59ed267a2b19cc28aac9ee7119914000ca478234af8d56893a025ae72cc647 `
>> `logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale" `
>> `wgengine.NewUserspaceEngine(tun "tailscale0") ... `
>> `wgengine.NewUserspaceEngine(tun "tailscale0") error: creating router: could 
>> not get iptables version: fork/exec /usr/bin/iptables: no such file or 
>> directory flushing log. `
>> `logger closing down `
>> `createEngine: creating router: could not get iptables version: fork/exec 
>> /usr/bin/iptables: no 

Re: [systemd-devel] Setting up a VPN daemon as a Portable Service

[Arian van Putten]

Afaik Portable services run in an isolated root and dont have access to the
hosts rootfs.  You'd have go include iptables and all its dependencies in
the portable services directory. If you don't want to do that you'd have to
use BindReadOnlyPaths= to give the service access to the required host
paths or you'd have to use a system extension.

That's probably why they advice running as a system extension.

I think there are mechanisms for setting up system extensions on startup
but I'm not familiar enough with the details. Maybe someone else in the
list knows.




On Thu, 6 Oct 2022, 20:21 Duncan Gibson,  wrote:

> Hi, everyone.
>
> The high-level overview: I'm trying to install Tailscale
>  as a portable service on my Steam Deck.
>
> Tailscale is a point-to-point VPN service, essentially a wrapper around
> Wireguard that helps with network setup and management. The Steam Deck is
> Valve's handheld PC running SteamOS 3, which is derived from Arch. It uses
> an A/B partition system for system files, meaning you can't install a
> service the normal way.
>
> There *is* a guide to do this ,
> posted on their own blog, but it uses system extensions which aren't good
> for services that you want to run on startup. Indeed, following that guide
> puts me in a state where I have to manually start the daemon every time I
> reboot my Deck, even with the service enabled.
>
> Let's move on to how I've started to do this.
>
> Tailscale is available through most package managers, but they also
> publish static binaries with systemd unit files
> .
>
> This script grabs that binary, extracts it, and moves it into a portable
> service directory structure.
>
> # download and extract Tailscale
> tarball="$(curl -s 'https://pkgs.tailscale.com/stable/?mode=json' | jq -r
> .Tarballs.amd64)"
> version="$(echo ${tarball} | cut -d_ -f2)"
> tar_dir="$(echo ${tarball} | cut -d. -f1-3)"
> curl -s "https://pkgs.tailscale.com/stable/${tarball}; -o tailscale.tgz
> tar xzf tailscale.tgz
> test -d $tar_dir
>
> # Set up our target directory structure
> mkdir -p
> tailscaled/{usr/{bin,sbin,lib/systemd/system},etc,proc,sys,dev,run,/var/tmp}
>
> # Copy tailscale-distributed files to the right place
> cp -rf $tar_dir/tailscaled tailscaled/usr/sbin/tailscaled
> cp -rf $tar_dir/systemd/tailscaled.service
> tailscaled/usr/lib/systemd/system/tailscaled.service
>
> # Write service os-release file
> source /etc/os-release
> cp -rf /etc/os-release tailscaled/etc/os-release
>
> Not automated yet is patching the provided unit file - you need to remove
> the EnvironmentFile line and "--port $PORT $FLAGS" options, and add
> [Exec]
> Environment="PATH=/usr/bin"
>
> Attach the portable service: sudo portablectl attach ./tailscaled
> --profile=trusted
> and try starting it: sudo systemctl start tailscaled
>
> It fails, leaving this in the logs:
>
> logtail started
> Program starting: v1.30.2-t24c524c78-gc399ae6fa, Go 1.19.1-tsb13188dd36:
> []string{"/usr/sbin/tailscaled",
> "--state=/var/lib/tailscale/tailscaled.state",
> "--socket=/run/tailscale/tailscaled.sock"}
> LogID: 0f59ed267a2b19cc28aac9ee7119914000ca478234af8d56893a025ae72cc647
> logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
> wgengine.NewUserspaceEngine(tun "tailscale0") ...
> wgengine.NewUserspaceEngine(tun "tailscale0") error: creating router:
> could not get iptables version: fork/exec /usr/bin/iptables: no such file
> or directory flushing log.
> logger closing down
> createEngine: creating router: could not get iptables version: fork/exec
> /usr/bin/iptables: no such file or directory
>
> iptables is, in fact, at /usr/bin/iptables, so what am I missing? Before I
> added the Environment line, I was getting errors that iptables wasn't on
> the PATH, so I suspect that now tailscaled can *see* iptables, but
> systemd isn't letting tailscaled run it.
>
> Thanks for having a look at this.
>