Re: [Duplicity-team] Stable snap broken? (was Re: Duplicity 0.8.09 Released)

[Kenneth Loafman]

I'm up for centralizing the discussion on anything but email.  Since a lot
of the discussion has already gone on in issue #119
, let's centralize
there for now.  It has the ability to do limited threading and markdown.

As for building on GitLab, I've tried.  Look back in history for
.gitlab-ci.yml and you'll see the build steps.  They stopped working on
core20 and I went back to a Ubuntu 18.04 VM build.  I would love to be able
to build on GitLab again, but things just don't work.

...Ken

On Mon, Mar 21, 2022 at 4:16 PM  wrote:

> On 21.03.2022 22:05, Aaron wrote:
> > On 2022-03-21 20:50, Aaron wrote:
> >
> >> I feel like the "proper" solution here is to add a step to the CI/CD
> process on Gitlab to test the snap before it goes to stable. That is a bit
> awkward as snaps do not run inside Docker containers, which are what is
> easiest in Gitlab CI/CD. I will try to make some time to investigate this
> -- maybe I can set up a spend-limited Linode API key and just have it run
> some smoke tests on the snap as part of a specified release process or
> something. To do it all as a pipeline, I think we would have to do the
> release and snapping in there as well.
> >>
> > As much to help me find it again in the future, I see that I found some
> guidance on building snaps on Gitlab and pushing them to the snap store
> here:
>
> how about we move this discussion into a ticket on gitlab, so it get's
> archived for the future? :)
>
> >
> https://forum.snapcraft.io/t/building-and-pushing-snap-packages-from-gitlab/9537
> <
> https://forum.snapcraft.io/t/building-and-pushing-snap-packages-from-gitlab/9537
> >
> >
> > It's a bit old now, but hopefully still useful. Perhaps we can now use
> remote build to avoid faffing with special containers for building?
>
> having done some remote building, i'd say f***ing slow but very useful for
> non amd64 archs. having said that. snapping locally in a CI will probably
> still be the fastest, although personally i do build and push releases
> manually, always, just to keep things under control. it's not like i
> release daily or so ;)
>
> ..ede
>
>
> ___
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : duplicity-team@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~duplicity-team
Post to : duplicity-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~duplicity-team
More help   : https://help.launchpad.net/ListHelp

Re: [Duplicity-team] Stable snap broken? (was Re: Duplicity 0.8.09 Released)

[edgar . soldin]

On 21.03.2022 22:05, Aaron wrote:

On 2022-03-21 20:50, Aaron wrote:


I feel like the "proper" solution here is to add a step to the CI/CD process on 
Gitlab to test the snap before it goes to stable. That is a bit awkward as snaps do not 
run inside Docker containers, which are what is easiest in Gitlab CI/CD. I will try to 
make some time to investigate this -- maybe I can set up a spend-limited Linode API key 
and just have it run some smoke tests on the snap as part of a specified release process 
or something. To do it all as a pipeline, I think we would have to do the release and 
snapping in there as well.


As much to help me find it again in the future, I see that I found some 
guidance on building snaps on Gitlab and pushing them to the snap store here:


how about we move this discussion into a ticket on gitlab, so it get's archived 
for the future? :)


https://forum.snapcraft.io/t/building-and-pushing-snap-packages-from-gitlab/9537 


It's a bit old now, but hopefully still useful. Perhaps we can now use remote 
build to avoid faffing with special containers for building?


having done some remote building, i'd say f***ing slow but very useful for non 
amd64 archs. having said that. snapping locally in a CI will probably still be 
the fastest, although personally i do build and push releases manually, always, 
just to keep things under control. it's not like i release daily or so ;)

..ede


___
Mailing list: https://launchpad.net/~duplicity-team
Post to : duplicity-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~duplicity-team
More help   : https://help.launchpad.net/ListHelp

Re: [Duplicity-team] Stable snap broken? (was Re: Duplicity 0.8.09 Released)

[edgar . soldin]

and while this autoupdating may be nice, it is really risky for production 
systems. so once again. let's adopt snap versioning like blender does it (see 
below and https://snapcraft.io/blender).

let's keep 'latest/{stable..edge}' but additionally push at least 
'/stable', so users can decide to (re)install fixed versions too. 
makes downgrading in case something goes wrong so much easier.

:$ snap info blender
name:  blender
summary:   Blender is the free and open source 3D creation suite.
SNIP
channels:
  latest/stable: 3.1.0   2022-03-09 (1925) 225MB classic
  latest/candidate:  ↑
  latest/beta:   3.1.0   2022-02-11 (1731) 226MB classic
  latest/edge:   ↑
  3.1/stable:3.1.0   2022-03-09 (1925) 225MB classic
  3.1/candidate: 3.1.0   2022-03-21 (2020) 225MB classic
  3.1/beta:  ↑
  3.1/edge:  ↑
  3.0/stable:3.0.1   2022-01-26 (1653) 223MB classic
  3.0/candidate: ↑
  3.0/beta:  ↑
  3.0/edge:  ↑
  2.93lts/stable:2.93.8  2022-02-02 (1698) 205MB classic
  2.93lts/candidate: 2.93.9  2022-03-21 (2019) 205MB classic
  2.93lts/beta:  ↑
  2.93lts/edge:  ↑
  2.92/stable:   2.92.0  2021-02-25  (111) 196MB classic
  2.92/candidate:↑
  2.92/beta: ↑
  2.92/edge: ↑
  2.91/stable:   2.91.2  2021-01-20   (65) 193MB classic
  2.91/candidate:↑
  2.91/beta: ↑
  2.91/edge: ↑
  2.90/stable:   2.90.1  2020-11-24   (47) 187MB classic

On 21.03.2022 21:50, Aaron wrote:

Kenneth and team,
On Mon, Jan 13, 2020 at 5:50 AM Aaron mailto:li...@whitehouse.kiwi.nz>> wrote:

[...]

If it would be helpful, I would be happy to volunteer to be more involved 
in the testing and release process of snaps through channels to give a second 
pair of eyes/small amount more QA.

We could do something with the snap channels like the following:

 1. Reserve Edge for bleeding edge like automatically created snaps from 
Master.
 2. You (Kenneth) put new releases into Beta.
 3. I could run some basic tests (I have the beginnings of some scripts 
that run tests in bare lxc containers to catch missing undeclared dependencies 
etc) and, if all looks good, promote them from Beta into Candidate.
 4. We could leave these sitting in Candidate for at least a week or so. If 
I know these have had some initial tests then I'm happy to switch some of my 
boxes to this track. Hopefully others on the team will, too. Then we will catch 
any major issues (like this) before any Stable users.
 5. When/If you think appropriate, you can advance any candidates to Stable.

Up to you, of course. My intention is to keep you in control of both adding 
new snaps and releasing to stable, but giving one additional set of checks/pair 
of eyes before hitting anyone's production boxes. I think Snaps are a great 
distribution mechanism for us, but it does mean there's no distro 
testing/QA/beta process between us and our users and we need to take a bit more 
of that responsibility.

[...]


On 2020-01-13 17:23, Kenneth Loafman wrote:


Aaron,
I agree with everything you said!
I'll start pushing to /candidate/ instead of /stable/ for releases. You can 
check it out and promote it to stable.  Snaps may lag behind repo releases, but 
that's OK.  One change I've made is that now snaps are versioned with the revno 
of the repo, like 0.8.10rev1548 so we can identify them better.

[...]



I am resurrecting this old thread because it looks to me as though the stable snap 
broke again (https://gitlab.com/duplicity/duplicity/-/issues/119 
 -- it also broke for me, 
looks like the last two days and also back on 8 and 9 March, so I suspect rev 243 and 
250). I would like to try to do more to help solve this problem, so I am not 
criticising -- I know you have asked for help and I have not offered to do any more 
and you have been left carrying all the work.

I feel like the "proper" solution here is to add a step to the CI/CD process on 
Gitlab to test the snap before it goes to stable. That is a bit awkward as snaps do not 
run inside Docker containers, which are what is easiest in Gitlab CI/CD. I will try to 
make some time to investigate this -- maybe I can set up a spend-limited Linode API key 
and just have it run some smoke tests on the snap as part of a specified release process 
or something. To do it all as a pipeline, I think we would have to do the release and 
snapping in there as well.

A while back you summarised your build/release process for the list here:

https://lists.launchpad.net/duplicity-team/msg04839.html 


Would you mind please setting out what you do now, including the snapping part? 
I see there is a makesnap, testsnap and pushsnap in the tools folder, for 
example.

I actually have my main server tracking Release Candidate instead of Stable as 
mentioned above, so I likely would have

Re: [Duplicity-team] Stable snap broken? (was Re: Duplicity 0.8.09 Released)

[Kenneth Loafman]

OK, snap is in candidate as 0.8.10.1558.  It's been checked with
snapcraft.io release-tools.

...Ken


On Mon, Jan 13, 2020 at 11:23 AM Kenneth Loafman 
wrote:

> Aaron,
>
> I agree with everything you said!
>
> I'll start pushing to *candidate* instead of *stable* for releases. You
> can check it out and promote it to stable.  Snaps may lag behind repo
> releases, but that's OK.  One change I've made is that now snaps are
> versioned with the revno of the repo, like 0.8.10rev1548 so we can identify
> them better.
>
> I'm fighting with versioning right now.  The source is unversioned in the
> repository and I want to change that so that we can track it better.  Some
> of the package managers for other distro's are not pulling from the release
> tarballs, instead pulling from repo and not running *dist/makedist*, so
> no version when running '*duplicity --version'* .
>
> I'm looking at the module *pkg_resources*.  It seems a bit heavy, but
> it's the module-de-jour for packaging.  I hope it has traction.
>
> Any thoughts?
>
> ...Thanks,
> ...Ken
>
>
>
>
>
>
> On Mon, Jan 13, 2020 at 5:50 AM Aaron  wrote:
>
>> Kenneth,
>>
>> On 2020-01-09 18:25, Kenneth Loafman wrote:
>>
>> Gave up and swapped to Python3 for snaps.  It works now.
>>
>> As to what was happening, I don't know, I can't reproduce it.
>>
>>
>> Great that you turned around a fix so quickly. Are we happy enough with
>> Python 3 for backends etc that we want to throw that straight out into
>> Stable or should we keep stable as the 0.8.08 version while we beta test
>> the Python 3 snap? I suppose we've at least had some testing of duplicity
>> on Python 3 from the distros dropping Python 2.
>>
>> If it would be helpful, I would be happy to volunteer to be more involved
>> in the testing and release process of snaps through channels to give a
>> second pair of eyes/small amount more QA.
>>
>> We could do something with the snap channels like the following:
>>
>>1. Reserve Edge for bleeding edge like automatically created snaps
>>from Master.
>>2. You (Kenneth) put new releases into Beta.
>>3. I could run some basic tests (I have the beginnings of some
>>scripts that run tests in bare lxc containers to catch missing undeclared
>>dependencies etc) and, if all looks good, promote them from Beta into
>>Candidate.
>>4. We could leave these sitting in Candidate for at least a week or
>>so. If I know these have had some initial tests then I'm happy to switch
>>some of my boxes to this track. Hopefully others on the team will, too.
>>Then we will catch any major issues (like this) before any Stable users.
>>5. When/If you think appropriate, you can advance any candidates to
>>Stable.
>>
>> Up to you, of course. My intention is to keep you in control of both
>> adding new snaps and releasing to stable, but giving one additional set of
>> checks/pair of eyes before hitting anyone's production boxes. I think Snaps
>> are a great distribution mechanism for us, but it does mean there's no
>> distro testing/QA/beta process between us and our users and we need to take
>> a bit more of that responsibility.
>>
>> Thanks for your work fixing this.
>>
>> Kind regards,
>>
>> Aaron
>>
>>
>>
>>
___
Mailing list: https://launchpad.net/~duplicity-team
Post to : duplicity-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~duplicity-team
More help   : https://help.launchpad.net/ListHelp

Re: [Duplicity-team] Stable snap broken? (was Re: Duplicity 0.8.09 Released)

[Kenneth Loafman]

Aaron,

I agree with everything you said!

I'll start pushing to *candidate* instead of *stable* for releases. You can
check it out and promote it to stable.  Snaps may lag behind repo releases,
but that's OK.  One change I've made is that now snaps are versioned with
the revno of the repo, like 0.8.10rev1548 so we can identify them better.

I'm fighting with versioning right now.  The source is unversioned in the
repository and I want to change that so that we can track it better.  Some
of the package managers for other distro's are not pulling from the release
tarballs, instead pulling from repo and not running *dist/makedist*, so no
version when running '*duplicity --version'* .

I'm looking at the module *pkg_resources*.  It seems a bit heavy, but it's
the module-de-jour for packaging.  I hope it has traction.

Any thoughts?

...Thanks,
...Ken






On Mon, Jan 13, 2020 at 5:50 AM Aaron  wrote:

> Kenneth,
>
> On 2020-01-09 18:25, Kenneth Loafman wrote:
>
> Gave up and swapped to Python3 for snaps.  It works now.
>
> As to what was happening, I don't know, I can't reproduce it.
>
>
> Great that you turned around a fix so quickly. Are we happy enough with
> Python 3 for backends etc that we want to throw that straight out into
> Stable or should we keep stable as the 0.8.08 version while we beta test
> the Python 3 snap? I suppose we've at least had some testing of duplicity
> on Python 3 from the distros dropping Python 2.
>
> If it would be helpful, I would be happy to volunteer to be more involved
> in the testing and release process of snaps through channels to give a
> second pair of eyes/small amount more QA.
>
> We could do something with the snap channels like the following:
>
>1. Reserve Edge for bleeding edge like automatically created snaps
>from Master.
>2. You (Kenneth) put new releases into Beta.
>3. I could run some basic tests (I have the beginnings of some scripts
>that run tests in bare lxc containers to catch missing undeclared
>dependencies etc) and, if all looks good, promote them from Beta into
>Candidate.
>4. We could leave these sitting in Candidate for at least a week or
>so. If I know these have had some initial tests then I'm happy to switch
>some of my boxes to this track. Hopefully others on the team will, too.
>Then we will catch any major issues (like this) before any Stable users.
>5. When/If you think appropriate, you can advance any candidates to
>Stable.
>
> Up to you, of course. My intention is to keep you in control of both
> adding new snaps and releasing to stable, but giving one additional set of
> checks/pair of eyes before hitting anyone's production boxes. I think Snaps
> are a great distribution mechanism for us, but it does mean there's no
> distro testing/QA/beta process between us and our users and we need to take
> a bit more of that responsibility.
>
> Thanks for your work fixing this.
>
> Kind regards,
>
> Aaron
>
>
>
>
___
Mailing list: https://launchpad.net/~duplicity-team
Post to : duplicity-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~duplicity-team
More help   : https://help.launchpad.net/ListHelp

Re: [Duplicity-team] Stable snap broken? (was Re: Duplicity 0.8.09 Released)

[Aaron]

Kenneth, 

On 2020-01-09 18:25, Kenneth Loafman wrote:

> Gave up and swapped to Python3 for snaps.  It works now. 
> 
> As to what was happening, I don't know, I can't reproduce it.

Great that you turned around a fix so quickly. Are we happy enough with
Python 3 for backends etc that we want to throw that straight out into
Stable or should we keep stable as the 0.8.08 version while we beta test
the Python 3 snap? I suppose we've at least had some testing of
duplicity on Python 3 from the distros dropping Python 2. 

If it would be helpful, I would be happy to volunteer to be more
involved in the testing and release process of snaps through channels to
give a second pair of eyes/small amount more QA. 

We could do something with the snap channels like the following: 

* Reserve Edge for bleeding edge like automatically created snaps from
Master.
* You (Kenneth) put new releases into Beta.
* I could run some basic tests (I have the beginnings of some scripts
that run tests in bare lxc containers to catch missing undeclared
dependencies etc) and, if all looks good, promote them from Beta into
Candidate.
* We could leave these sitting in Candidate for at least a week or so.
If I know these have had some initial tests then I'm happy to switch
some of my boxes to this track. Hopefully others on the team will, too.
Then we will catch any major issues (like this) before any Stable users.
* When/If you think appropriate, you can advance any candidates to
Stable.

Up to you, of course. My intention is to keep you in control of both
adding new snaps and releasing to stable, but giving one additional set
of checks/pair of eyes before hitting anyone's production boxes. I think
Snaps are a great distribution mechanism for us, but it does mean
there's no distro testing/QA/beta process between us and our users and
we need to take a bit more of that responsibility. 

Thanks for your work fixing this. 

Kind regards, 

Aaron___
Mailing list: https://launchpad.net/~duplicity-team
Post to : duplicity-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~duplicity-team
More help   : https://help.launchpad.net/ListHelp

Re: [Duplicity-team] Stable snap broken? (was Re: Duplicity 0.8.09 Released)

[Kenneth Loafman]

Gave up and swapped to Python3 for snaps.  It works now.

As to what was happening, I don't know, I can't reproduce it.

...Ken


On Thu, Jan 9, 2020 at 10:29 AM Kenneth Loafman  wrote:

> Found the problem, but don't know of a fix yet.
>
> Snaps run under Python2.7 and it's dying in duplicity/dup_time.py with:
>
> >>> from duplicity import dup_time
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "duplicity/dup_time.py", line 67, in 
> page for more information.""")
> TypeError: 'str' object is not callable
>
> Line 76 is a long message in dup_time using _(), which is supposed to be
> installed globally through duplicity/__init__.py.
>
> Essentially, _() is not working anymore.
>
> On Wed, Jan 8, 2020 at 7:33 PM Kenneth Loafman 
> wrote:
>
>> OK, reverted to rev 33, 0.8.08.  Will check it out tomorrow.
>>
>> There were no changes to the yaml file and the snap build went fine.
>> Very odd!  :-(
>>
>> ...Thanks,
>> ...Ken
>>
>>
>> On Wed, Jan 8, 2020 at 5:07 PM Aaron  wrote:
>>
>>> Kenneth,
>>>
>>> The 0.8.09 (45) snap seems completely broken for me on two machines
>>> (16.04 and 19.04):
>>>
>>> $ duplicity --version
>>> Traceback (most recent call last):
>>>File "/snap/duplicity/45/bin/duplicity", line 31, in 
>>>  from future import standard_library
>>> ModuleNotFoundError: No module named 'future'
>>>
>>> Reverting to 0.8.08 (sudo snap revert duplicity) works as expected.
>>>
>>> If this is broken for others we need to fix this in the store asap --
>>> I'm tracking the stable channel on a real box and this completely broke
>>> it when it auto-updated last night.
>>>
>>> We need to be super-sensitive about putting things in the stable channel
>>> that are not production ready as the blessing and the curse of snaps is
>>> that they keep themselves up to date with the latest thing in the
>>> channel without user interaction. Thankfully that also should mean we
>>> can fix it quickly by just reverting stable to 0.8.08.
>>>
>>> Kind regards,
>>>
>>> Aaron
>>>
>>>
>>> ___
>> Mailing list: https://launchpad.net/~duplicity-team
>> Post to : duplicity-team@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~duplicity-team
>> More help   : https://help.launchpad.net/ListHelp
>>
>
___
Mailing list: https://launchpad.net/~duplicity-team
Post to : duplicity-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~duplicity-team
More help   : https://help.launchpad.net/ListHelp

Re: [Duplicity-team] Stable snap broken? (was Re: Duplicity 0.8.09 Released)

[Kenneth Loafman]

Found the problem, but don't know of a fix yet.

Snaps run under Python2.7 and it's dying in duplicity/dup_time.py with:

>>> from duplicity import dup_time
Traceback (most recent call last):
  File "", line 1, in 
  File "duplicity/dup_time.py", line 67, in 
page for more information.""")
TypeError: 'str' object is not callable

Line 76 is a long message in dup_time using _(), which is supposed to be
installed globally through duplicity/__init__.py.

Essentially, _() is not working anymore.

On Wed, Jan 8, 2020 at 7:33 PM Kenneth Loafman  wrote:

> OK, reverted to rev 33, 0.8.08.  Will check it out tomorrow.
>
> There were no changes to the yaml file and the snap build went fine.  Very
> odd!  :-(
>
> ...Thanks,
> ...Ken
>
>
> On Wed, Jan 8, 2020 at 5:07 PM Aaron  wrote:
>
>> Kenneth,
>>
>> The 0.8.09 (45) snap seems completely broken for me on two machines
>> (16.04 and 19.04):
>>
>> $ duplicity --version
>> Traceback (most recent call last):
>>File "/snap/duplicity/45/bin/duplicity", line 31, in 
>>  from future import standard_library
>> ModuleNotFoundError: No module named 'future'
>>
>> Reverting to 0.8.08 (sudo snap revert duplicity) works as expected.
>>
>> If this is broken for others we need to fix this in the store asap --
>> I'm tracking the stable channel on a real box and this completely broke
>> it when it auto-updated last night.
>>
>> We need to be super-sensitive about putting things in the stable channel
>> that are not production ready as the blessing and the curse of snaps is
>> that they keep themselves up to date with the latest thing in the
>> channel without user interaction. Thankfully that also should mean we
>> can fix it quickly by just reverting stable to 0.8.08.
>>
>> Kind regards,
>>
>> Aaron
>>
>>
>> ___
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : duplicity-team@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~duplicity-team
Post to : duplicity-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~duplicity-team
More help   : https://help.launchpad.net/ListHelp

Re: [Duplicity-team] Stable snap broken? (was Re: Duplicity 0.8.09 Released)

[Kenneth Loafman]

OK, reverted to rev 33, 0.8.08.  Will check it out tomorrow.

There were no changes to the yaml file and the snap build went fine.  Very
odd!  :-(

...Thanks,
...Ken


On Wed, Jan 8, 2020 at 5:07 PM Aaron  wrote:

> Kenneth,
>
> The 0.8.09 (45) snap seems completely broken for me on two machines
> (16.04 and 19.04):
>
> $ duplicity --version
> Traceback (most recent call last):
>File "/snap/duplicity/45/bin/duplicity", line 31, in 
>  from future import standard_library
> ModuleNotFoundError: No module named 'future'
>
> Reverting to 0.8.08 (sudo snap revert duplicity) works as expected.
>
> If this is broken for others we need to fix this in the store asap --
> I'm tracking the stable channel on a real box and this completely broke
> it when it auto-updated last night.
>
> We need to be super-sensitive about putting things in the stable channel
> that are not production ready as the blessing and the curse of snaps is
> that they keep themselves up to date with the latest thing in the
> channel without user interaction. Thankfully that also should mean we
> can fix it quickly by just reverting stable to 0.8.08.
>
> Kind regards,
>
> Aaron
>
>
>
___
Mailing list: https://launchpad.net/~duplicity-team
Post to : duplicity-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~duplicity-team
More help   : https://help.launchpad.net/ListHelp