Re: classic 32 bit application

[James Henstridge]

On 31 March 2017 at 05:38, Seth Arnold  wrote:
> On Thu, Mar 30, 2017 at 08:10:26AM +0200, Alistair Grant wrote:
>> I'm trying to package a 32 bit software development environment: Pharo
>> Smalltalk (http://pharo.org).
>>
>> I've got it working OK as a devmode package, but as soon as I switch it
>> to classic confinement it fails to run.
>
> I was under the impression the usual progression is from devmode to
> strict. Are you certain classic is the correct direction for your snap?

I ran into a similar conundrum for the Python snap I built.  If your
package contains a language runtime and interactive shell, it is
difficult to decide what sort of confinement policy makes sense.  It
is possible to run under strict confinement with few or any interfaces
connected, but depending on what the user wants to do might want a lot
more permission (e.g. ability to access the network, ability to write
to the home directory, etc).

At present the best option seems to be to package things with strict
confinement but ensure that it will be functional if installed with
--classic.  That gives safety by default, but full functionality on
request.  Of course, this means snapcraft isn't giving any help with
the necessary link flags to get things working reliably on non-Ubuntu
systems.  I guess that's something to try and solve next.

James.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Experimental Python interpreter snap

[James Henstridge]

On 21 February 2017 at 23:37, Barry Warsaw <ba...@ubuntu.com> wrote:
> On Feb 21, 2017, at 09:30 PM, James Henstridge wrote:
>
>>So we might be able to do a single package that can both serve as a
>>runtime for other snaps and as a useful Python development
>>environment.
>
> It would be interesting to see, but my tendency is to want separate
> interpreter environments for different purposes.  See my previous post re: a
> locked down system interpreter for /usr/bin scripts.
>
> The problem with one-size-fits-all (and we have this problem today with
> deb-based /usr/bin/python{2,3}) is that people sudo pip install all kinds of
> crazy things into their {site,dist}-packages, and that can break things, which
> are difficult to debug (though we're adding some useful features to 3.7 to
> help with that).


That's the thing: a confined app using the interpreter from my package
via the content interface is effectively going to get its own
environment.  It won't be searching for Python modules under /usr
because that's not where the Python installation comes from.  It won't
search for modules installed to ~/.local/lib/pythonX.Y because it
won't have permission to read that directory.

So even if we get it so "sudo pip install" works, it won't actually
have any impact on other snaps using the interpreter.

>
> So I think it makes some sense to separate these concerns: OS platform use,
> confined snap application use, developer playground.  Virtualenvs are the
> typical "Pythonic" way of doing that, but snaps provide another opportunity
> for confinement.
>
> (Of course "/usr/bin/python{,2,3}" is the long-established ui for that
> developer playground.)

Yep.  So I think it probably makes most sense for the Python runtime
snap to default to classic confinement so that it behaves as a user
would expect for interactive/development work, with pip ready to
install to ~/.local/lib/..., or to the system wide $SNAP_DATA folder
if the user really wants to install things system wide.  This would
seem to satisfy both use cases well.

James

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Experimental Python interpreter snap

[James Henstridge]

On 21 February 2017 at 20:53, Stuart Bishop <stuart.bis...@canonical.com> wrote:
> On 21 February 2017 at 18:35, James Henstridge
> <james.henstri...@canonical.com> wrote:
>
>>> You could probably also get the pip in your snap to install packages
>>> to $SNAP_USER_DATA or $SNAP_DATA if run as root. Although most devs
>>> would stick to using virtualenvs outside of the snap for this,
>>> assuming a modern enough Python.
>>
>> With the snap as it stands, it is most useful as a runtime for other
>> snaps rather than for interactive use or for development.  If you
>> install my package with --devmode to disable confinement, it could be
>> useful for development, but there isn't really an opportunity for a
>> shared site-packages directory ($SNAP_DATA for the python snap won't
>> be accessible to other snaps).
>
> I think I was thinking that the main snap could use classic
> confinement, allowing you to use it as the interpreter for scripts
> located anywhere. And snaps using the interface would remain contained
> as they are. Assuming we are allowed to mix interfaces and classic
> confinement :)

As I understand it, this should be fine: when you connect two snaps
with the content interface, all it does is bind mounts a directory
from the slot-side snap to a mount point within the plug-side snap.
It shouldn't matter what confinement policies the two snaps are using.
And since the snap data is a read-only squashfs, neither side can
alter the content that is being shared.


> But now I think on it further, its probably not a good idea to pollute
> the main python snap when it is being used as a dependency.

Well, it should be pretty easy to detect whether we're running in the
context of the Python snap: checking if sys.prefix is equal to $SNAP
would probably be enough (from a different snap, the interpreter will
be mounted somewhere else).  It wouldn't be difficult to add extra
directories to sys.path in this case, and with confinement disabled I
suspect we'd pick up the standard "user site" directory automatically.

So we might be able to do a single package that can both serve as a
runtime for other snaps and as a useful Python development
environment.

James.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Experimental Python interpreter snap

[James Henstridge]

On 21 February 2017 at 16:57, Stuart Bishop <stuart.bis...@canonical.com> wrote:
> On 20 February 2017 at 11:41, James Henstridge
> <james.henstri...@canonical.com> wrote:
>> On 20 February 2017 at 10:45, XiaoGuo Liu <xiaoguo@canonical.com> wrote:
>>> Hi James,
>>>
>>> Nice. This is a nice example showing how to reduce a python snap package.  A
>>> few days ago, I also made a small example to make use of the python3 coming
>>> with the core at:
>>>
>>> https://github.com/liu-xiao-guo/httpstat
>>>
>>> In the above example, I in fact do not package the python. It works.
>>
>> So I guess the main differences are that this makes it easy to use a
>> newer version of Python than existed at the time Xenial was released.
>> I suspect I could also build a Python 2.7 snap using the same
>> techniques, if you want to go in the other direction.
>
> Yes. I love the idea of being able to use 3.6 or 3.7, and not have to
> rebuild and release my snaps every time there is a point release.
>
> I also like the idea of having easy access to nightly builds of the
> development branch, and antique versions. I think having old versions
> packaged  in snaps is nicer than the current approach of  the
> deadsnakes ppa.

It probably wouldn't be too difficult to build a snap tracking 3.7
development, yeah.


>> The Python in my snap is also set up to automatically use packages
>> included in your own snap without fiddling with environment variables
>> or sys.path.
>
> You could probably also get the pip in your snap to install packages
> to $SNAP_USER_DATA or $SNAP_DATA if run as root. Although most devs
> would stick to using virtualenvs outside of the snap for this,
> assuming a modern enough Python.

With the snap as it stands, it is most useful as a runtime for other
snaps rather than for interactive use or for development.  If you
install my package with --devmode to disable confinement, it could be
useful for development, but there isn't really an opportunity for a
shared site-packages directory ($SNAP_DATA for the python snap won't
be accessible to other snaps).

And if you are trying to package up a Python app that will be running
under snap confinement, do you want writable locations in the default
sys.path?

Maybe there is room for some compromise: we could enable a
site-packages dir in $SNAP_DATA/$SNAP_USER_DATA when running the
interpreter under the python snap's context, but not when running
under a different package's confinement.

James.

>
> --
> Stuart Bishop <stuart.bis...@canonical.com>
>
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/snapcraft

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Experimental Python interpreter snap

[James Henstridge]

On 20 February 2017 at 10:41, Spencer  wrote:
> I thought a main feature of snaps was to include all dependencies so that 
> they couldn't be changed out from underneath a package.  For example, my 
> script was written for 3.6, but would be incompatible with a future release, 
> say 4.0.

I've included two content interface slots on my snap: one with the
content ID "python3" and the other with the content ID "python3.6".
The idea being that a program that doesn't care about getting new
versions (e.g. my trivial hello world snap) could use the first, while
ones that really want python 3.6.x (e.g. if they contain compiled
extensions) could request the second.

I don't think there is any point in distinguishing micro releases
though, since the Python core developers have a good track record when
it comes to releasing security/bug fix updates to their previous
releases.


> Still, the ability to share a dependency like the Python interpreter among 
> python snaps may be a good idea if there are zillions of Python snaps.
>
> At my work, we got tired of everyone maintaining their own local python 
> installation, because we all ended up with slightly different versions of 
> various python modules installed.  Worse, some modules installed for some 
> while not for others.  So we started tracking a shared Python installation in 
> git.  One problem we found is that Python is not relocatable.  This showed up 
> when people cloned our repository in a different place.  Are you sure that 
> your snapped Python is relocatable?  If so, I'd like to know how that works.  
> Is the $ORIGIN variable standard?

Note that the the only things being shared here is the Python
interpreter and the standard library.  If one snap includes a bunch of
third party Python packages in their $SNAP/lib/python3.6/site-packages
directory, they won't be visible to a second snap that has also used
the interface.  The effect is quite similar to the isolation you get
from virtualenv.

As far as relocatability goes, part of it is provided by Python
proper.  Here's sys.path from the interpreter provided by the
python36-jamesh snap:

$ python36-jamesh.python3
Python 3.6.0 (default, Feb 20 2017, 01:27:20)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/snap/python36-jamesh/7/lib/python36.zip',
'/snap/python36-jamesh/7/lib/python3.6',
'/snap/python36-jamesh/7/lib/python3.6/lib-dynload']

And here it is when running in the context of my hello-world snap:

$ snap run --shell hello-world
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

$ $SNAP/python/bin/python3
Python 3.6.0 (default, Feb 20 2017, 01:27:20)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/snap/hello-world/x1/python/lib/python36.zip',
'/snap/hello-world/x1/python/lib/python3.6',
'/snap/hello-world/x1/python/lib/python3.6/lib-dynload']

What I added was configuring DT_RUNPATH for the executable and
extensions.  This augments the set of directories the dynamic linker
searches for shared library dependencies.  If a directory in this list
contains the token "$ORIGIN", it will be expanded to the the directory
containing the program or library.  So by including "$ORIGIN/../lib"
in the runpath for bin/python3.6, I can make sure it will find the
libpython in the directory next to it, no matter where it happens to
be bind mounted.  You can find more information about this in the
ld.so(8) man page.

James.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Experimental Python interpreter snap

[James Henstridge]

To learn a bit more about I put together a snap for Python 3.6.0,
which can be installed with:

snap install --edge python36-jamesh

You can then run "python36-jamesh.python3", which will give you the a
Python shell running with strict confinement, with the full standard
library available.

Now I know Snapcraft already has support for packaging Python
applications, so what benefits does this package add?  There were a
few extra points in how I built the package:

1. the interpreter binary and extension modules all have appropriate
$ORIGIN relative rpath set.

2. a sitecustomize.py is provided that will add
$SNAP/lib/python3.6/site-packages to sys.path. (more on why this is
useful later)

This makes the interpreter fully relocatable in the file system while
still being able to find the bundled libraries.  In turn, this means
the interpreter is functional when exported to another snap via the
content interface.

To demonstrate this, I put together a trivial "hello world" snapcraft
project here:

https://github.com/jhenstridge/python-snap-pkg/tree/master/examples/hello-world

After building this package, it can be run after installing and
connecting the interface:

$ snap install --dangerous hello-world_0.1_amd64.snap
hello-world 0.1 installed
$ snap connect hello-world:python3 python36-jamesh:python3
$ hello-world
Hello world!

Since the hello-world snap doesn't actually include Python, it is
quite light weight (4 kB, which I think is as small as a squashfs
gets).  The space savings may not be that great with a single snap
(the interpreter snap is almost 20MB), but the space savings increase
as you install more snaps using the interface.  It also means that we
could upgrade to Python 3.6.1 (when it comes out) without rebuilding
this snap.

And since the interpreter is being run under the hello-world snap's
confinement policy, it can do potentially do things the main
"python36-jamesh.python3" binary can't.  For example, if you add the
"network" plug, you'll be able to access the network.

And the sitecustomize script will also mean the interpreter can locate
packages shipped in the plug snap.

I'd be interested in any suggestions or feedback about the snap.

Thanks,

James.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: ubuntu-app-platform updated to Qt 5.6.2

[James Henstridge]

On 9 February 2017 at 10:43, XiaoGuo Liu  wrote:
> I am now trying to snap a html5 webapp. My source code is here at:
>
> https://github.com/liu-xiao-guo/wuziqi

Looking at the snapcraft.yaml file, you don't seem to be using the
launcher shell script provided by the part.  Without that,
LD_LIBRARY_PATH and other environment variables won't be set up
correctly to use libraries provided by ubuntu-app-platform.

James.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Max retries exceeded?

[James Henstridge]

On 6 February 2017 at 11:47, Michael Nelson
 wrote:
> On Mon, Feb 6, 2017 at 1:25 PM Michi Henning 
> wrote:
>>
>> I keep getting this error when doing a snapcraft cleanbuild. Any
>> suggestions? I checked name resolution, and my DNS resolves
>> parts.snapcraft.io just fine.
>>
>
> Hi Michi
>
> Have you verified your DNS resolution and/or curling from within your
> container? That is:
>
>>
>> Thanks,
>>
>> Michi.
>>
>> Setting up python3-lxml (3.5.0-1build1) ...
>> Setting up python3-petname (2.0-0ubuntu1~16.04) ...
>> Setting up python3-progressbar (2.3-2) ...
>> Setting up python3-pymacaroons (0.9.2-0ubuntu1) ...
>> Setting up python3-requests-toolbelt (0.6.0-2) ...
>> Setting up python3-simplejson (3.8.1-1ubuntu2) ...
>> Setting up python3-tabulate (0.7.5-1) ...
>> Setting up python3-xdg (0.25-4) ...
>> Setting up xdelta3 (3.0.8-dfsg-1ubuntu2) ...
>> Setting up python3-libarchive-c (2.1-3) ...
>> Setting up python3-magic (1:5.25-2ubuntu1) ...
>> Setting up snapcraft (2.26) ...
>> Processing triggers for libc-bin (2.23-0ubuntu5) ...
>> HTTPSConnectionPool(host='parts.snapcraft.io', port=443): Max retries
>> exceeded with url: /v1/parts.yaml (Caused by
>> NewConnectionError('> object at 0x7f5537c29ba8>: Failed to establish a new connection: [Errno -3]
>> Temporary failure in name resolution',))
>> Stopping snapcraft-merely-model-ewe
>> Command '['lxc', 'exec', 'snapcraft-merely-model-ewe', '--', 'snapcraft',
>> 'snap', '--output', 'thumbnailer_2.4_amd64.snap']' returned non-zero exit
>> status 1
>>
>>
>
> Can you try:
> $ lxc exec snapcraft-merely-model-ewe -- curl
> https://parts.snapcraft.io/v1/parts.yaml
>
> ? I can only confirm that curling that from various machines here responds
> fine (and that there haven't been any alerts for that service for the past
> day).

I was able to reproduce Michi's result.  I reran the command as
"snapcraft cleanbuild -d", which drops to a shell within the container
on the failure.  Using "getent hosts $hostname", I can get an IP
address for archive.ubuntu.com, but not parts.snapcraft.io.

Running strace, I disected the DNS communication here:
http://paste.ubuntu.com/23938750/ -- it is performing a query with
recursion desired, but the response only comes back with the CNAME
record and no A records.  If I perform the same request against
Google's public DNS (8.8.8.8), I get the CNAME and a bunch of A
records.

I'm guessing this is coming from the system resolver on the host
system.  I'm on zesty, so this could be a systemd-resolve problem.

James.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Snap security questions

[James Henstridge]

Hi,

On our team we've been working to snap the thumbnailer project.  While
there are some problems that are probably specific to this package,
there were a few that I suspect might affect other packages too:

1. Intra-snap D-Bus communication

The thumbnailer D-Bus service exposes a number of methods that were
intended for use by the "thumbnailer-admin" helper program, but not by
outside clients.  The generic "dbus" snappy interface does a good job
of making sure clients can't call these privileged methods, but I'm
left needing a way to get thumbnailer-admin working again.

I noticed that the default AppArmor rules allow communication via unix
domain sockets with other apps from the same snap, so one easy way to
solve my problem would be to also allow applications to send and
receive arbitrary messages over the session bus to/from other
applications from the same snap.  This would let me get
thumbnailer-admin working without having to expose the same abilities
to third party snaps through a slot.

I filed a bug about this one here:

https://bugs.launchpad.net/snappy/+bug/1659724


2. Use of the libapparmor aa_is_enabled and aa_query_label APIs

When deciding whether to do work on behalf of a client,
thumbnailer-service uses a couple libapparmor API calls to determine
whether the client has access to a file.  Neither of these are working
under snappy confinement.

The first call we use is aa_is_enabled(), but it seems the policy is
to strict to let us determine whether AppArmor is enabled or not.

Next we use aa_query_label() to perform the file access check.  This
fails when trying to read /proc/$pid/mounts to determine where
securityfs is mounted.  If that is fixed, it will likely fail when
trying to access the "/sys/kernel/security/apparmor/.access" file
within.

I've filed a bug for this one here:

https://bugs.launchpad.net/snappy/+bug/1660957


3. QNetworkAccessManager wants to access NetworkManager

We use QNetworkAccessManager as our HTTP library.  This results in a
number of denials for D-Bus method calls to NetworkManager.

I can make these denials go away by plugging in to the
":network-manager" slot, but a quick look at that interface shows that
it grants a lot more permissions than I need or want (i.e. permission
to reconfigure the network).

I imagine that a lot of snaps will use QNetworkAccessManager, so it
would be nice if the calls it makes were allowed by some
auto-connectable interface.  If not as part of ":network", then
something similar.


James.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft