Re: Post mortem on today's packaging error.

2019-02-12 Thread Carlton Gibson
Hi All, 

Thanks for all the comments. 

Cloning to a tmp folder and checking out a specific commit is both quick 
and easy. That's not a bad idea: it should eliminate the point of failure 
that occurred here. 
(The `git clean` step should do that too, we use the `fdx` flags to 
eliminate ignored files etc, and it does remove build and dist dirs, but... 
well... something went wrong.) 

I want to add automated running of the test suite against the generated 
src-dist. That would have caught the issues too. This seems to me to be the 
most reliable way to verify that what got packaged was in fact correct. 

I don't think we need to automate any further the final actual shipping of 
the package. `twine upload -s` is pretty easy, and I do worry about us 
putting the signing keys on even a private server really. 

Grrr. Definitely sub-par yesterday. Onwards! :)
C.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ac3e4168-139c-4973-a1fc-e6e1e3b3a45e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Post mortem on today's packaging error.

2019-02-12 Thread Luke Plant

Hi Carlton,

I have had problems in the past with `setup.py sdist` and friends 
actually depending on existing artefacts in build directories from 
previous runs. A quick google turns up this bug report - 
https://github.com/pypa/setuptools/issues/436 - originally filed by  
oh, me :-)  and it is still open. It seems that `git clean -dfx` does 
clean out these files, so adding that to the release Makefile should fix 
the issue if this was the root problem. The release process document 
needs to be adjusted to mention that wanted build artefacts need to be 
moved somewhere else before preparing the next release, otherwise they 
are going to get deleted before they are uploaded.


Luke

On 11/02/2019 19:28, Carlton Gibson wrote:

Hi all.

This morning I released four versions of Django. Three of which, for 
2.1, 2.0 and 1.11. (i.e. all the actually supported versions) were 
broken.
In the package were additional files from `master`/2.2 which shouldn't 
have been there.


This afternoon I have released follow-ups to correct this issue.

First of all, sorry about that, and for any inconvenience caused.


Then, these are process issues so, how can we do better next time?

I'm not 100% sure what occurred.

* The history in Git is correct.
* I must have had the right commits checked out, because the package 
metadata is correct. (Filenames, version numbers and so on.)


My best guess is that I've failed to `git clean` correctly before 
building each release.
I'm not certain here because switching between branches doesn't leave 
the repos in an unclean state, and I'm pretty sure it was clean, but 
this seems the most likely error.



Q: is there a nice git command to "assert I'm at exactly this tag"?


Steps I've taken:

* Moved the `git clean` step into the helper script used to build the 
packages. No chance of then missing it.
* Added a `diff`-step after building just to make sure what's in the 
`django` module matches the checkout.
The second of these, whilst just a visual check, would have worked 
with what ended up in the package vs what was in my working tree when 
I checked later, but I'm not sure it would have caught the issue when 
the package was created (because presumably my working tree was wrong 
at that point).


A similar issue affects the checksums, which check that nothing 
changed since it was packaged, but not that the right things were 
packaged...

(Similarly, pip install worked without error.)

On the new 1.11 package, I created a virtualenv with Python 2 and ran 
the test suite. This worked so it should be good. But it would be good 
to automate this.
(I'll look into it.) We don't ship the tests in the wheel, so we have 
to use the src-dist for this. (We could then diff the two django 
modules to make sure they were the same.)


On the other packages I visually looked for incorrect files, but, 
beyond specific migrations (etc) that were known bad from the previous 
release, this doesn't generalise.


"zaytsev" on IRC suggested running `makemigrations --check`, so having 
a test project to install against might also be worth it.

(I'd guess running the test suite would be sufficient...)


Any other thoughts very welcome.


Kind Regards,

Carlton

--
You received this message because you are subscribed to the Google 
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-developers+unsubscr...@googlegroups.com 
.
To post to this group, send email to 
django-developers@googlegroups.com 
.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/90c531fc-3d5b-4d9e-a55f-b88f2e04bb54%40googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c795f471-9f0b-8794-8899-ed3bbec821aa%40cantab.net.
For more options, visit https://groups.google.com/d/optout.


Re: Post mortem on today's packaging error.

2019-02-12 Thread Aleksi Häkli
Jazzband (https://jazzband.co/about/releases) uses an approach that builds 
and pushes the PyPI packages to an intermediate repository that is owned by 
the Jazzband organization.

The Jazzband intermediate repository then allows publishing them from the 
Jazzband organization to PyPI via a push-button deployment from the Travis 
build. 

This eliminates the need for having the public PyPI warehouse credentials 
published to Travis or other target, but requires setting up a private 
repository.

An approach like this (2) of course introduces two additional components to 
the trust chain compared to the current model (1):

1. With a simple PyPI upload, only the PyPI warehouse and the uploader has 
to be ultimately trusted, and package signatures are easy to check against 
known public PGP keys with 2 parties of trust, but
2. with an intermediate private PyPI upload from e.g. Travis. both Travis 
and the private intermediate server have to be trusted in addition to PyPI 
warehouse and the original author with 4 parties of trust.

On Tuesday, 12 February 2019 09:36:09 UTC+2, Florian Apolloner wrote:
>
>
>
> On Monday, February 11, 2019 at 11:01:55 PM UTC+1, Adam Johnson wrote:
>>
>> Jamesie’s suggestion to use CI is also valid but a bunch more work. I 
>> guess the main advantage is you get a blank slate container to work in, 
>> which a fresh checkout to a temp dir provides most of the gain for less 
>> work.
>>
>
> If I remember we have been hesitant in the past because that would require 
> us to give credentials to PyPi etc to the CI service. That said I think 
> that is a risk we could take.
>
> Cheers,
> Florian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a1f957d4-87c4-4f09-87cc-d8fefa6c5c98%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Post mortem on today's packaging error.

2019-02-11 Thread Florian Apolloner


On Monday, February 11, 2019 at 11:01:55 PM UTC+1, Adam Johnson wrote:
>
> Jamesie’s suggestion to use CI is also valid but a bunch more work. I 
> guess the main advantage is you get a blank slate container to work in, 
> which a fresh checkout to a temp dir provides most of the gain for less 
> work.
>

If I remember we have been hesitant in the past because that would require 
us to give credentials to PyPi etc to the CI service. That said I think 
that is a risk we could take.

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fb0d567d-d01a-44d9-983a-ac0a1ba93de1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Post mortem on today's packaging error.

2019-02-11 Thread Adam Johnson
Thanks for the detailed post mortem Carlton.

Andrew’s suggested approach to do at least a checkout to a fresh directory
makes sense to me. Even got checkout and clean aren’t enough to bring an
exisiting checkout folder to the same state as git won’t touch files in the
gitignore. Note you can do a checkout or a local repo to a different
directory, you wouldn’t need to reclone from a different remote.

Jamesie’s suggestion to use CI is also valid but a bunch more work. I guess
the main advantage is you get a blank slate container to work in, which a
fresh checkout to a temp dir provides most of the gain for less work.

That’s my two cents

On Mon, 11 Feb 2019 at 17:30, Jamesie Pic  wrote:

> Hi Carlton !
>
> Seems like you're having as much fun as I had when doing releases
> manually :D Just sharing some food for thought here.
>
> Nowadays I have it automated and rely on setupmeta to keep myself away
> from touching setup.py, and just have to push git tags :
> http://github.com/zsimic/setupmeta
>
> Then a silly little script like that to publish to pypi :
> https://yourlabs.io/oss/shyml/blob/master/shyml/sh.yml
>
> All python modules on yourlabs.io/oss have this kind of setup ... of
> course you're still stuck on jenkins so I can understand it's not as
> flexible as say gitlab-ci or drone-ci, but there's probably also a
> way.
>
> Hope this helps
>
> Keep up the good work, you'll make it !
>
> Have a great day
>
> --
> ∞
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAC6Op1_rgn8BnHHS-0%2B05yMKFi5g%2Bq%2BKmHZVz1-SP-sgxY-%3D3Q%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>
-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM3EUDHDCD_97cyYL-p772Pnpry2eKPK6%3D0dvOD%3DxnOBNA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Post mortem on today's packaging error.

2019-02-11 Thread Jamesie Pic
Hi Carlton !

Seems like you're having as much fun as I had when doing releases
manually :D Just sharing some food for thought here.

Nowadays I have it automated and rely on setupmeta to keep myself away
from touching setup.py, and just have to push git tags :
http://github.com/zsimic/setupmeta

Then a silly little script like that to publish to pypi :
https://yourlabs.io/oss/shyml/blob/master/shyml/sh.yml

All python modules on yourlabs.io/oss have this kind of setup ... of
course you're still stuck on jenkins so I can understand it's not as
flexible as say gitlab-ci or drone-ci, but there's probably also a
way.

Hope this helps

Keep up the good work, you'll make it !

Have a great day

-- 
∞

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op1_rgn8BnHHS-0%2B05yMKFi5g%2Bq%2BKmHZVz1-SP-sgxY-%3D3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Post mortem on today's packaging error.

2019-02-11 Thread Andrew Godwin
I also ran into this several times while releasing Channels - several
releases had the wrong files ship out in them due to Git weirdness.

In the end, my solution was to build the release artifacts over on Travis
to guarantee a fresh build environment each time, but I doubt that would
work for Django. Adding "git clean" to the script is probably good enough,
but I might be tempted to make a script where you pass it a commit hash and
it checks it out to a fresh temporary directory and packages from there?

Andrew

On Mon, Feb 11, 2019 at 8:28 AM Carlton Gibson 
wrote:

> Hi all.
>
> This morning I released four versions of Django. Three of which, for 2.1,
> 2.0 and 1.11. (i.e. all the actually supported versions) were broken.
> In the package were additional files from `master`/2.2 which shouldn't
> have been there.
>
> This afternoon I have released follow-ups to correct this issue.
>
> First of all, sorry about that, and for any inconvenience caused.
>
>
> Then, these are process issues so, how can we do better next time?
>
> I'm not 100% sure what occurred.
>
> * The history in Git is correct.
> * I must have had the right commits checked out, because the package
> metadata is correct. (Filenames, version numbers and so on.)
>
> My best guess is that I've failed to `git clean` correctly before building
> each release.
> I'm not certain here because switching between branches doesn't leave the
> repos in an unclean state, and I'm pretty sure it was clean, but this seems
> the most likely error.
>
>
> Q: is there a nice git command to "assert I'm at exactly this tag"?
>
>
> Steps I've taken:
>
> * Moved the `git clean` step into the helper script used to build the
> packages. No chance of then missing it.
> * Added a `diff`-step after building just to make sure what's in the
> `django` module matches the checkout.
>
> The second of these, whilst just a visual check, would have worked with
> what ended up in the package vs what was in my working tree when I checked
> later, but I'm not sure it would have caught the issue when the package was
> created (because presumably my working tree was wrong at that point).
>
> A similar issue affects the checksums, which check that nothing changed
> since it was packaged, but not that the right things were packaged...
> (Similarly, pip install worked without error.)
>
> On the new 1.11 package, I created a virtualenv with Python 2 and ran the
> test suite. This worked so it should be good. But it would be good to
> automate this.
> (I'll look into it.) We don't ship the tests in the wheel, so we have to
> use the src-dist for this. (We could then diff the two django modules to
> make sure they were the same.)
>
> On the other packages I visually looked for incorrect files, but, beyond
> specific migrations (etc) that were known bad from the previous release,
> this doesn't generalise.
>
> "zaytsev" on IRC suggested running `makemigrations --check`, so having a
> test project to install against might also be worth it.
> (I'd guess running the test suite would be sufficient...)
>
>
> Any other thoughts very welcome.
>
>
> Kind Regards,
>
> Carlton
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/90c531fc-3d5b-4d9e-a55f-b88f2e04bb54%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqvLhhEoB2WFy%3DC4KWA2gLns%3DadhCGwfnYidJsHOjVpUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Post mortem on today's packaging error.

2019-02-11 Thread Carlton Gibson
Hi all. 

This morning I released four versions of Django. Three of which, for 2.1, 
2.0 and 1.11. (i.e. all the actually supported versions) were broken. 
In the package were additional files from `master`/2.2 which shouldn't have 
been there. 

This afternoon I have released follow-ups to correct this issue. 

First of all, sorry about that, and for any inconvenience caused. 


Then, these are process issues so, how can we do better next time? 

I'm not 100% sure what occurred. 

* The history in Git is correct. 
* I must have had the right commits checked out, because the package 
metadata is correct. (Filenames, version numbers and so on.) 

My best guess is that I've failed to `git clean` correctly before building 
each release. 
I'm not certain here because switching between branches doesn't leave the 
repos in an unclean state, and I'm pretty sure it was clean, but this seems 
the most likely error.


Q: is there a nice git command to "assert I'm at exactly this tag"?  


Steps I've taken: 

* Moved the `git clean` step into the helper script used to build the 
packages. No chance of then missing it. 
* Added a `diff`-step after building just to make sure what's in the 
`django` module matches the checkout. 
 
The second of these, whilst just a visual check, would have worked with 
what ended up in the package vs what was in my working tree when I checked 
later, but I'm not sure it would have caught the issue when the package was 
created (because presumably my working tree was wrong at that point). 

A similar issue affects the checksums, which check that nothing changed 
since it was packaged, but not that the right things were packaged...
(Similarly, pip install worked without error.) 

On the new 1.11 package, I created a virtualenv with Python 2 and ran the 
test suite. This worked so it should be good. But it would be good to 
automate this. 
(I'll look into it.) We don't ship the tests in the wheel, so we have to 
use the src-dist for this. (We could then diff the two django modules to 
make sure they were the same.) 

On the other packages I visually looked for incorrect files, but, beyond 
specific migrations (etc) that were known bad from the previous release, 
this doesn't generalise. 

"zaytsev" on IRC suggested running `makemigrations --check`, so having a 
test project to install against might also be worth it. 
(I'd guess running the test suite would be sufficient...) 


Any other thoughts very welcome. 


Kind Regards,

Carlton

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/90c531fc-3d5b-4d9e-a55f-b88f2e04bb54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.