Re: Ubuntu Core: how the file-system works

[Luca Dionisi]

On Wed, Jan 25, 2017 at 3:16 PM, Jamie Strandboge  wrote:
> The security policy changes are merged in master and you will be able to
> manipulate rt_tables by connecting the network-control interface in the 
> upcoming
> snapd 2.22.

Wooohooo!

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Jamie Strandboge]

On Mon, 2017-01-23 at 21:30 +0100, Luca Dionisi wrote:
> On Mon, Jan 23, 2017 at 6:28 PM, Jamie Strandboge  wrote:
> > 
> > I will be looking at the security policy side of this so if you can, please
> > comment in the bug what specific commands you are using in your snap for
> > using
> > rt_tables so I can repeat tham and make sure they are supported.
> Done.
> 
Thanks!

The security policy changes are merged in master and you will be able to
manipulate rt_tables by connecting the network-control interface in the upcoming
snapd 2.22.

-- 
Jamie Strandboge | http://www.canonical.com



signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Luca Dionisi]

On Mon, Jan 23, 2017 at 6:28 PM, Jamie Strandboge  wrote:
> I will be looking at the security policy side of this so if you can, please
> comment in the bug what specific commands you are using in your snap for using
> rt_tables so I can repeat tham and make sure they are supported.

Done.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Jamie Strandboge]

On Mon, 2017-01-23 at 17:17 +0100, Luca Dionisi wrote:
> Hi all
> 
> I see that the issue has been taken care of. I will immediately
> download a daily-build image and check that the file rt_tables
> is writeable.
> 
> I haven't built a snap for my app yet, so I cannot test for the
> moment if my needs are all already fitted in the builtin
> interfaces. I will try as soon as I can to craft a snap in devmode
> and afterwards in strict mode. But I can't predict time.

I will be looking at the security policy side of this so if you can, please
comment in the bug what specific commands you are using in your snap for using
rt_tables so I can repeat tham and make sure they are supported.

[1]https://bugs.launchpad.net/snappy/+bug/1658298

-- 
Jamie Strandboge | http://www.canonical.com



signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Luca Dionisi]

On Mon, Jan 23, 2017 at 5:26 PM, Gustavo Niemeyer  wrote:
> It's definitely possible. It's just not very convenient yet.
>
> For tests, easiest might be to bind mount your modifications at runtime.

Ok, thanks. bind mount.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Gustavo Niemeyer]

It's definitely possible. It's just not very convenient yet.

For tests, easiest might be to bind mount your modifications at runtime.

On Mon, Jan 23, 2017 at 2:17 PM, Luca Dionisi 
wrote:

> Hi all
>
> I see that the issue has been taken care of. I will immediately
> download a daily-build image and check that the file rt_tables
> is writeable.
>
> I haven't built a snap for my app yet, so I cannot test for the
> moment if my needs are all already fitted in the builtin
> interfaces. I will try as soon as I can to craft a snap in devmode
> and afterwards in strict mode. But I can't predict time.
>
> Bear with me if I ask again, cause maybe the question went off the
> radar: is it possible to build an image where the rootfs has been
> modified a bit from what is in 'core' by default?
>
> --Luca
>
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/snapcraft
>



-- 

gustavo @ http://niemeyer.net
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Luca Dionisi]

Hi all

I see that the issue has been taken care of. I will immediately
download a daily-build image and check that the file rt_tables
is writeable.

I haven't built a snap for my app yet, so I cannot test for the
moment if my needs are all already fitted in the builtin
interfaces. I will try as soon as I can to craft a snap in devmode
and afterwards in strict mode. But I can't predict time.

Bear with me if I ask again, cause maybe the question went off the
radar: is it possible to build an image where the rootfs has been
modified a bit from what is in 'core' by default?

--Luca

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Gustavo Niemeyer]

Yes, it seems fine for network-control to allow it.

On Mon, Jan 23, 2017 at 10:17 AM, Oliver Grawert  wrote:

> hi,
> Am Samstag, den 21.01.2017, 11:33 +0100 schrieb Luca Dionisi:
> > On Fri, Jan 20, 2017 at 6:43 PM, Oliver Grawert 
> > wrote:
> > >
> > > yes, please start by filing it under the snappy umbrella [1]
> > > project
> > > and we'll add the necessary bug tasks for all bits and pieces
> > > involved
> > > then.
> > Done.
> >
> > While I wait for it to be fixed, is there a way to build a custom
> > Ubuntu Core image where I can change by myself some bits of the
> > rootfs?
>
> as mentioned in the bug ( http://pad.lv/1658298 ) the dir is now
> writable in the daily edge images (and in the respective core snap, on
> a stable image you can just "snap refresh --edge core" to get the
> updated version)...
>
> we still need to discuss if route manipulation should have its own
> interface or be a part of the network-control interface though.
>
> ciao
> oli
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
> an/listinfo/snapcraft
>
>


-- 
gustavo @ http://niemeyer.net
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Oliver Grawert]

hi,
Am Samstag, den 21.01.2017, 11:33 +0100 schrieb Luca Dionisi:
> On Fri, Jan 20, 2017 at 6:43 PM, Oliver Grawert 
> wrote:
> > 
> > yes, please start by filing it under the snappy umbrella [1]
> > project
> > and we'll add the necessary bug tasks for all bits and pieces
> > involved
> > then.
> Done.
> 
> While I wait for it to be fixed, is there a way to build a custom
> Ubuntu Core image where I can change by myself some bits of the
> rootfs?

as mentioned in the bug ( http://pad.lv/1658298 ) the dir is now
writable in the daily edge images (and in the respective core snap, on
a stable image you can just "snap refresh --edge core" to get the
updated version)... 

we still need to discuss if route manipulation should have its own
interface or be a part of the network-control interface though.

ciao
oli

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Gustavo Niemeyer]

Hi Luca,

If you look under interfaces/builtin in the source code of snapd, you'll
find some familliar names if you list *network* and *firewall* in there. I
suspect that what you want is very easy to fix by simply introducing an
additional apparmor entry in the right interface.

If you want to test something locally sooner rather than later (say,
today), then that would be best as it'd also tell us if the fix we'll
commit into the tree will actually work for you.

Otherwise, we can do that on our end and you just let us know if it worked
for your case or if you need additional permissions which are lacking an
interface for.



On Sat, Jan 21, 2017 at 8:33 AM, Luca Dionisi 
wrote:

> On Fri, Jan 20, 2017 at 6:43 PM, Oliver Grawert  wrote:
> > yes, please start by filing it under the snappy umbrella [1] project
> > and we'll add the necessary bug tasks for all bits and pieces involved
> > then.
>
> Done.
>
> While I wait for it to be fixed, is there a way to build a custom
> Ubuntu Core image where I can change by myself some bits of the
> rootfs?
>
> --Luca
>
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/snapcraft
>



-- 
gustavo @ http://niemeyer.net
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Luca Dionisi]

On Fri, Jan 20, 2017 at 6:43 PM, Oliver Grawert  wrote:
> yes, please start by filing it under the snappy umbrella [1] project
> and we'll add the necessary bug tasks for all bits and pieces involved
> then.

Done.

While I wait for it to be fixed, is there a way to build a custom
Ubuntu Core image where I can change by myself some bits of the
rootfs?

--Luca

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Oliver Grawert]

hi,
Am Freitag, den 20.01.2017, 18:15 +0100 schrieb Luca Dionisi:
> 
> I think I got it. So I will continue to use g_spawn_async_with_pipes
> in my code. But I will prepare the snap file so that when installed
> on
> Ubuntu Core (or snap based system) it will work even in confined
> mode.
> Also, it will have the exact version of the userspace tools that I
> will
> choose to ship.

exactly, use g_spawn_async_with_pipes but allow it to accept env vars
for the path of the executable it calls, that way you can create a
snapcraft.yaml that pulls in iptables and your command will just do the
right thing if $SNAP is set (i.e. iptables in your snap might live in
$SNAP/usr/sbin/ while on a deb system it is just /usr/sbin, this also
gives you full control over the iptables version in snappy)

 
> 
> 
> 
> So, going back to my first issue. I would like to be able to create
> routing table names on Ubuntu Core. To my knowledge this should be
> done by writing to the rt_tables file, and it is currently
> impossible.
> Should I consider filing a bug?
> 
yes, please start by filing it under the snappy umbrella [1] project
and we'll add the necessary bug tasks for all bits and pieces involved
then.

ciao
oli

[1] https://bugs.launchpad.net/snappy/+filebug

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: file system signatures and trust model, was Re: Ubuntu Core: how the file-system works

[George Gross]

thank you for the "how to" pointer to make custom Ubuntu Core images,
I'll stay tuned for replies on the other Qs...

   george


On Fri, 2017-01-20 at 16:01 +0100, Oliver Grawert wrote:
> hi,
> Am Freitag, den 20.01.2017, 09:43 -0500 schrieb George Gross:
> > Hi,
> > 
> > at the risk of wading into the weeds, you mentioned below that:
> > 
> > "...it also has the advantage that the core and kernel snaps are
> > signed
> > readonly squashfses and can not just be modified which adds a great
> > amount of extra security."
> > 
> > Is there a Wiki or document explaining the signature private key's
> > life
> > cycle management? For example, what process happens when the key
> > expires
> > or is compromised? Who is the entity that actually *signs* the file
> > system?
> 
> this is probably something the security and store teams can answer
> better than me.
> 
> > 
> > If you built a custom kernel and/or device drivers, how would your
> > binaries interact with this file system signature's verification? Can
> > you substitute your own software factory/store's signature?
> 
> you would create a complete own image based on your own developer
> signature using a signed model assertion.
> 
> https://docs.ubuntu.com/core/en/guides/build-device/image-building
> 
> has details on this.
> 
> > 
> > If you operate your own private CA and sign some file objects within
> > the
> > snap, does that CA need to be cross-certified with the trust anchor
> > CA
> > that is vouching for the identity applying the core/kernel file
> > system
> > signature?
> 
> again something the store people are better suited to answer, i dont
> exactly know how the CA store side is set up here :)
> 
> ciao
>   oli



-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Luca Dionisi]

On Fri, Jan 20, 2017 at 5:33 PM, Mark Shuttleworth  wrote:
> On 20/01/17 08:03, Luca Dionisi wrote:
>> If I understand it correctly, an unconfined app will be able in the
>> system
>> to do whatever my standard user would be able to. For instance, if I
>> log into my ubucore16 (the name of my KVM instance) and issue:
>>  sudo sysctl net.ipv4.ip_forward=1
>> -or-
>>  sudo ip address add 10.0.0.10 dev eth0
>> it reports success. Thus, if I run an unconfined app which tries to do the
>> same it will succeed. Whilst a strictly confined app would not, if it is
>> not hooked to a certain capability.
>> So far, so good?
>
> Ubuntu Core is confined-snaps-only. Ubuntu Classic allows less confined
> snaps.
>
> The commands you're wanting to run should be fine, though, with the
> right interfaces in place for your confined snap on Ubuntu Core. I think
> you meant that when you said 'hooked for a certain capability'. The
> devmode confinement should also be a useful workaround in your
> development process.

Ok.

On Fri, Jan 20, 2017 at 5:18 PM, Oliver Grawert  wrote:
> the firewall interface gives you access to the kernel firewall
> features, your snap would ship the necessary user space tools for this
> and run them in the snap. the interface will be the same on all snap
> based systems (pretty much like ufw builds in iptables, ipset in the
> snap [1])

I think I got it. So I will continue to use g_spawn_async_with_pipes
in my code. But I will prepare the snap file so that when installed on
Ubuntu Core (or snap based system) it will work even in confined mode.
Also, it will have the exact version of the userspace tools that I will
choose to ship.



So, going back to my first issue. I would like to be able to create
routing table names on Ubuntu Core. To my knowledge this should be
done by writing to the rt_tables file, and it is currently impossible.
Should I consider filing a bug?

--Luca

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Mark Shuttleworth]

On 20/01/17 08:03, Luca Dionisi wrote:
> If I understand it correctly, an unconfined app will be able in the
> system
> to do whatever my standard user would be able to. For instance, if I
> log into my ubucore16 (the name of my KVM instance) and issue:
>  sudo sysctl net.ipv4.ip_forward=1
> -or-
>  sudo ip address add 10.0.0.10 dev eth0
> it reports success. Thus, if I run an unconfined app which tries to do the
> same it will succeed. Whilst a strictly confined app would not, if it is
> not hooked to a certain capability.
> So far, so good?

Ubuntu Core is confined-snaps-only. Ubuntu Classic allows less confined
snaps.

The commands you're wanting to run should be fine, though, with the
right interfaces in place for your confined snap on Ubuntu Core. I think
you meant that when you said 'hooked for a certain capability'. The
devmode confinement should also be a useful workaround in your
development process.

Mark

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Oliver Grawert]

hi,
Am Freitag, den 20.01.2017, 17:03 +0100 schrieb Luca Dionisi:
> 
> > my first step here would be to use the default ubuntu-core image
> > and
> > start working on a snap you can install on top of it that uses the
> > existing interfaces. also take a look at the source of existing
> > snaps
> > for inspiration i.e. there is a "ufw" snap that does firewalling
> > that
> > should show how you can be able to manipulate iptables with
> > thefirewall-control interface.
> In my app I do network-control-related tasks by simply spawning
> standard
> linux commands. With g_spawn_async_with_pipes.
> 
> I don't use other "interfaces" and I would prefer not to use any
> particular interface that ties my program to one platform. Or do I
> misunderstand what you mean by "the firewall-control interface?"

the firewall interface gives you access to the kernel firewall
features, your snap would ship the necessary user space tools for this
and run them in the snap. the interface will be the same on all snap
based systems (pretty much like ufw builds in iptables, ipset in the
snap [1])

ciao
oli

[1] http://bazaar.launchpad.net/~jdstrand/ufw/trunk/view/head:/snapcraf
t.yaml

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Oliver Grawert]

hi,
Am Freitag, den 20.01.2017, 15:33 +0100 schrieb Luca Dionisi:
> 
> Since my "thingy" is going to sport an experimental routing protocol,
> I need
> to change some files on the fly. For instance /etc/iproute/rt_tables.
> Which
> I already see that is not writeable in my Ubuntu Core install.
> 
> Also I am going to use some commands that I haven't yet tested on
> Ubuntu
> Core. Mostly "ip" and "iptables", also in non-default network
> namespaces.
> And I don't know if they need internally write-access to some file.
> 
> Do you see anything about it that would be infeasible in a Ubuntu
> Core as
> it currently stands?
> 
> If not, what is my next step? Will I need to build a custom Ubuntu
> Core
> image? While testing, would I be able to remount the current image
> file
> system in read-write?
> 
you would not be able to mount the current rootfs in read-write, since
it is a squashfs file. 

for the above features you require ubuntu-core typically offers
interfaces that a snap can use to control specific parts of the system
[1]. i assume your goal is to have some app that can manage firewall,
routing and other networking aspects. 

my first step here would be to use the default ubuntu-core image and
start working on a snap you can install on top of it that uses the
existing interfaces. also take a look at the source of existing snaps
for inspiration i.e. there is a "ufw" snap that does firewalling that
should show how you can be able to manipulate iptables with
thefirewall-control interface. 

every time you hit a roadblock you file a bug and ask for extension of
the interface or for a new interface to be added (and indeed we also
accept patches if you already have an idea for a solution ;) ).
eventually you will end up with the system you desire.

snappy is a new approach so things you are used to from existing deb
based systems might work in different ways (like: you wont
make /etc/iproute/rt_tables writable but instead use the network-
control interface to manipulate the routing table).

ciao
oli

[1] http://snapcraft.io/docs/reference/interfaces


signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

file system signatures and trust model, was Re: Ubuntu Core: how the file-system works

[George Gross]

Hi,

at the risk of wading into the weeds, you mentioned below that:

"...it also has the advantage that the core and kernel snaps are signed
readonly squashfses and can not just be modified which adds a great
amount of extra security."

Is there a Wiki or document explaining the signature private key's life
cycle management? For example, what process happens when the key expires
or is compromised? Who is the entity that actually *signs* the file
system?

If you built a custom kernel and/or device drivers, how would your
binaries interact with this file system signature's verification? Can
you substitute your own software factory/store's signature?

If you operate your own private CA and sign some file objects within the
snap, does that CA need to be cross-certified with the trust anchor CA
that is vouching for the identity applying the core/kernel file system
signature?

tia,
George

On Fri, 2017-01-20 at 15:14 +0100, Oliver Grawert wrote:





-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Mark Shuttleworth]

On 20/01/17 06:33, Luca Dionisi wrote:
> Since my "thingy" is going to sport an experimental routing protocol,
> I need
> to change some files on the fly. For instance /etc/iproute/rt_tables. Which
> I already see that is not writeable in my Ubuntu Core install.
>
> Also I am going to use some commands that I haven't yet tested on Ubuntu
> Core. Mostly "ip" and "iptables", also in non-default network namespaces.
> And I don't know if they need internally write-access to some file.
>
> Do you see anything about it that would be infeasible in a Ubuntu Core as
> it currently stands?

That all sounds like stuff we would want to be doable. There are a
number of folks using Ubuntu Core for networking kit, so we are adding
capabilities for that domain all the time. We just recently added
network namespace control.

Generally, you want to think about how best to express config in a way
that easily survives upgrades. Editing something like rt_tables should
be fine. But where you have something that multiple pieces want to edit,
or a set of things which need to line up, we need to design a config
item which drives those things consistently so the device can *never*
end up in a broken state, by design. Make sense?

Mark




-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Oliver Grawert]

hi,
Am Freitag, den 20.01.2017, 14:59 +0100 schrieb Luca Dionisi:
> Hi all,
> 
> I am planning to build a raspberry-based gadget and I would rather
> use
> Ubuntu Core on it. So I am right now using it on a KVM in order to
> see
> how it works.
> 
> First of all I need to understand how the file-system works. Because
> I
> need to edit some system files.
> 
> My first question to the list is: why don't I see two partitions on
> the disk? I recall having read that Ubuntu Core was able to rollback
> to
> the previous version of "core" snap thanks to a second partition.
> Do I miss something?
> 
two readonly partitions were the 15.04 way when we still used image
based upgrades (a technology that was developed for the phone images)

with 16.04 snappy images switched to have everything as a snap this
includes kernel, bootloader (gadget) and the rootfs (core).

during boot the initrd mounts the readonly core snap (which is a
squashfs) and bind-mounts a few required files into writable
directories so they become writable (typically this are a bunch of
selected cache and config files for system services). 

rollback of kernel or core is done by switching back and forth between
the different revisions of the snaps now, not by hopping between
partitions any more, this way snappy can now use a single partition
(the snaps just sit on the writeable partition) which reduces the
complexity a lot. it also has the advantage that the core and kernel
snaps are signed readonly squashfses and can not just be modified which
adds a great amount of extra security.

ciao
oli

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Ubuntu Core: how the file-system works

[Mark Shuttleworth]

On 20/01/17 05:59, Luca Dionisi wrote:
> First of all I need to understand how the file-system works. Because I
> need to edit some system files.

Ubuntu Core is designed to offer a super-reliable and predictable
upgrade experience, so core system files are often fixed ("immutable").
It will be interesting to know what you need at the base level so we can
expose it as a standard config element.

> My first question to the list is: why don't I see two partitions on
> the disk? I recall having read that Ubuntu Core was able to rollback to
> the previous version of "core" snap thanks to a second partition.
> Do I miss something?

In 15.04 we had an A/B partition system. Now we have evolved to
something much better, which is mountable compressed filesystems. The
A/B filesystems are now single files on the base filesystem. That's
better because it lets us use space much more efficiently, or decide
later we want three or four versions instead of just A/B versions, etc.

Mark


-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft