Fellow Report - November 26, 2016

[Tim Graham]

Triaged

---

https://code.djangoproject.com/ticket/27520 - sqlmigrate generates a 
duplicate DROP INDEX statement (duplicate)

https://code.djangoproject.com/ticket/27483 - Add a login_required 
decorator for AJAX requests (wontfix)

https://code.djangoproject.com/ticket/27498 - Filtering annotated field in 
SQLite returns wrong results (accepted)

https://code.djangoproject.com/ticket/27521 - QueryDict is inconsistently 
converted to kwargs between CPython and PyPy (needsinfo)

https://code.djangoproject.com/ticket/27528 - No default object-level 
permissions provider (duplicate)

https://code.djangoproject.com/ticket/27538 - Value of JSONField is being 
re-encoded to string even though being already encoded (worksforme)

Reviewed/committed

--

https://github.com/django/django/pull/7583 - Completed django.dispatch test 
coverage.

https://github.com/django/django/pull/7370 - Fixed #27295 -- Added a system 
check to prohibit model names that start or end with an underscore or 
contain double underscores.

https://github.com/django/django/pull/7449 - Fixed #27250 -- Removed 'for 
="..."' from CheckboxSelectMultiple's . 

https://github.com/django/django/pull/7554 - Fixed #27221 -- Doc'd how to 
escape a percent symbol in ugettext().

https://github.com/django/django/pull/7594 - Added tests for 
contrib.sitemaps.ping_google().

https://github.com/django/django/pull/7577 - Fixed #27504 -- Allowed using 
the ORM after an error and rollback when autocommit is off.

https://github.com/django/django/pull/7571 - Fixed #24959 -- Fixed queries 
using negative timedeltas on MySQL and Oracle.

https://github.com/django/django/pull/7567 - Fixed #27499 -- Made 
Prefetches pickle without evaluating their QuerySet.

https://github.com/django/django/pull/7550 - Fixed #25966 -- Made 
get_user_model() work at import time.

https://github.com/django/django/pull/7523 - Fixed #24370 -- Recommended 
starting with a custom user model.

https://github.com/django/django/pull/7588 - Fixed #27517 -- Fixed charset 
param in SimpleTemplateResponse.__init__().
https://github.com/django/django/pull/7603 - Fixed #27532 -- Deprecated 
Model._meta.has_auto_field

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/435b4932-ed1f-4fd0-9ddf-3e2206c4ae73%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: automating Django releases

[Carl Meyer]

Hi Tim,

On 11/25/2016 11:49 AM, Tim Graham wrote:
> After doing releases about once a month for a while, I'm thinking it
> would be nice if releasing Django could be a bit more automated. As far
> as I know, the process hasn't changed too much in 10 years, while the
> state of Python packaging has improved.
> 
> Currently doing a release requires bumping the VERSION tuple in
> django/__init__.py [0], creating tarball/wheel and checksum files,
> asking someone in IRC to verify the checks look fine, and upload files
> to various locations [1]. My idea would be to use setuptools_scm [2]
> (that choice because this is the only tool I know about) to eliminate
> the need to bump/set a VERSION tuple and instead fetch the version
> information based on git tags. Something like (based on what I found in
> djanog-hosts [3]):
> 
> __version__ = pkg_resources.get_distribution('django').version

+1 for more automation, but I recommend simply automating the bumping of
the hardcoded VERSION in the source tree (should not be hard to do). I
recommend against relying on git for release versioning, and even more
strongly against depending on pkg_resources to get the right version.
It's just so much less robust and reliable than having a version
hardcoded into the source tree.

The above pkg_resources line only works if the current source tree is
also installed in the current Python environment. If you are working in
one source tree, but some other Django is installed, you will get the
wrong version. If you are working in a source tree you haven't
installed, you will get no version at all. We used to use a similar
pkg_resources-based scheme in pip, and it caused no end of trouble.

The git-based version fetching only works if you are working from a git
repo, not an exported source tree. An exported source tree, independent
of git, should still know at least what released version it represents.
I am sure if we break this, it will break somebody's workflow.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7fc5a22c-bd8c-5c4e-0d8d-dee495837d81%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: automating Django releases

[Josh Smeaton]

Appreciate the security concerns of the jenkins boxes, but Jenkins is the 
obvious choice as a task runner. My first thought would be to bring up a 
special slave that was particularly hardened and walled off from the 
regular PR runners. I'm unsure how far access to the slave from the master 
could be minimised via permissions etc. At that point though, it'd probably 
be easier to have a simple bash script on this make believe instance that 
ops could ssh to and run. Or a combination of both - using jenkins for the 
version bump and package build, pushing the artifact elsewhere, then 
manually running a sign + deploy step.

>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9c8d0288-85bb-4d02-96e2-1e018462b5de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: automating Django releases

[Aymeric Augustin]

I agree that most of the workflow could be scripted.

The manual verification step for checksums exists because there was an error in 
checksums once in the distant past and we took some flak for that. Since you’re 
making releases regularly, you’re much less likely to make a mistake. A script 
(hopefully) wouldn’t make mistakes either. So that step can be dropped.

To expand on what Florian said, as far as I know, the ops team treats Jenkins 
as an insecure machine. Since it contains only open-source software, there’s 
nothing to protect. It runs on an isolated domain (djangoci.com 
) to prevent a compromise from affecting 
djangoproject.com  services. Significant steps to 
increase its security would have to be taken before it could be trusted to 
generate and sign releases.

Best regards,

-- 
Aymeric.

> On 26 Nov 2016, at 11:02, Florian Apolloner  wrote:
> 
> Hi,
> 
> +1 to automating as much as possible; one thing though: Personally I do not 
> like uploading via Jenkins as a security issue there might compromise our 
> PyPi account. This is especially important if a build slave where able to do 
> something to the master… So I think as first step it would be nice to have a 
> script which automates everything we have locally and then if we want to, we 
> can still put it into Jenkins.
> 
> Cheers,
> Florian
> 
> On Friday, November 25, 2016 at 8:49:54 PM UTC+1, Tim Graham wrote:
> After doing releases about once a month for a while, I'm thinking it would be 
> nice if releasing Django could be a bit more automated. As far as I know, the 
> process hasn't changed too much in 10 years, while the state of Python 
> packaging has improved.
> 
> Currently doing a release requires bumping the VERSION tuple in 
> django/__init__.py [0], creating tarball/wheel and checksum files, asking 
> someone in IRC to verify the checks look fine, and upload files to various 
> locations [1]. My idea would be to use setuptools_scm [2] (that choice 
> because this is the only tool I know about) to eliminate the need to bump/set 
> a VERSION tuple and instead fetch the version information based on git tags. 
> Something like (based on what I found in djanog-hosts [3]):
> 
> __version__ = pkg_resources.get_distribution('django').version
> 
> One issue I'm not sure how to deal with is how to provides 
> backwards-compatibility for django.VERSION, the tuple format that looks 
> something like:
> 
> VERSION = (1, 11, 0, 'alpha', 0)
> 
> I think removing or deprecating it would be quite disruptive for the 
> community as I've seen lots of third-party apps that do checks like "if 
> django.VERSION < (1, 9): ...", for example. I suspect it's possible to parse 
> the version string to create something similar, but I wondered if anyone has 
> experience with that.
> 
> Finally, uploading to PyPI and djangoproject.com  
> (release and checksum files) could be done automatically by Jenkins or 
> something similar.
> 
> Thanks for your comments, concerns, and suggestions.
> 
> [0] 
> https://github.com/django/django/blob/6252fd6314f66d2d303cc47c791ffefd27169b42/django/__init__.py#L5
>  
> 
> [1] 
> https://docs.djangoproject.com/en/dev/internals/howto-release-django/#actually-rolling-the-release
>  
> 
> [2] https://github.com/pypa/setuptools_scm 
> 
> [3] 
> https://github.com/jazzband/django-hosts/blob/0eae77b31927ddb0267943d6f9bcfc22f14706f2/django_hosts/__init__.py#L11
>  
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/3a3a59e7-b4e4-4bc9-8538-67d040459840%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django 

Re: automating Django releases

[Florian Apolloner]

Hi,

+1 to automating as much as possible; one thing though: Personally I do not 
like uploading via Jenkins as a security issue there might compromise our 
PyPi account. This is especially important if a build slave where able to 
do something to the master… So I think as first step it would be nice to 
have a script which automates everything we have locally and then if we 
want to, we can still put it into Jenkins.

Cheers,
Florian

On Friday, November 25, 2016 at 8:49:54 PM UTC+1, Tim Graham wrote:
>
> After doing releases about once a month for a while, I'm thinking it would 
> be nice if releasing Django could be a bit more automated. As far as I 
> know, the process hasn't changed too much in 10 years, while the state of 
> Python packaging has improved.
>
> Currently doing a release requires bumping the VERSION tuple in 
> django/__init__.py [0], creating tarball/wheel and checksum files, asking 
> someone in IRC to verify the checks look fine, and upload files to various 
> locations [1]. My idea would be to use setuptools_scm [2] (that choice 
> because this is the only tool I know about) to eliminate the need to 
> bump/set a VERSION tuple and instead fetch the version information based on 
> git tags. Something like (based on what I found in djanog-hosts [3]):
>
> __version__ = pkg_resources.get_distribution('django').version
>
> One issue I'm not sure how to deal with is how to provides 
> backwards-compatibility for django.VERSION, the tuple format that looks 
> something like:
>
> VERSION = (1, 11, 0, 'alpha', 0)
>
> I think removing or deprecating it would be quite disruptive for the 
> community as I've seen lots of third-party apps that do checks like "if 
> django.VERSION < (1, 9): ...", for example. I suspect it's possible to 
> parse the version string to create something similar, but I wondered if 
> anyone has experience with that.
>
> Finally, uploading to PyPI and djangoproject.com (release and checksum 
> files) could be done automatically by Jenkins or something similar.
>
> Thanks for your comments, concerns, and suggestions.
>
> [0] 
> https://github.com/django/django/blob/6252fd6314f66d2d303cc47c791ffefd27169b42/django/__init__.py#L5
> [1] 
> https://docs.djangoproject.com/en/dev/internals/howto-release-django/#actually-rolling-the-release
> [2] https://github.com/pypa/setuptools_scm
> [3] 
> https://github.com/jazzband/django-hosts/blob/0eae77b31927ddb0267943d6f9bcfc22f14706f2/django_hosts/__init__.py#L11
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3a3a59e7-b4e4-4bc9-8538-67d040459840%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: automating Django releases

[Marc Tamlyn]

I'm pretty confident we can just automate the process you currently do.

On 26 November 2016 at 05:40, Josh Smeaton  wrote:

> I think automating as much of the release process is a fantastic idea.
> Regarding bumping the release numbers, have you seen the bumpversion
> project https://pypi.python.org/pypi/bumpversion?
>
> After some trial and error, I seem to have a configuration that works with
> the existing version tuple in django/__init__.py. Bumping the release
> number will change the VERSION tuple. The existing get_current_version will
> continue to work as normal, as will the setup.py. You can experiment with
> the tag/commit options and other options - the hard part is getting the
> serialize and parse options correct. This configuration goes in the
> existing setup.cfg.
>
> [bumpversion]
> current_version = (1, 11, 0, 'alpha', 0)  # you have to set this the first
> time, then bumpversion updates it along with the VERSION tuple in
> django/__init__.py
> commit = False
> tag = False
> serialize = ({major}, {minor}, {patch}, {prerel}, {prerelnum})
> parse = \((?P\d+),\s(?P\d+),\s(?P\d+),\s(?P<
> prerel>\'alpha\'|\'beta\'|\'rc\'),\s(?P\d+)\)
>
> [bumpversion:file:django/__init__.py]
>
> [bumpversion:part:prerel]
> values =
> 'alpha'
> 'beta'
> 'rc'
>
> And here's an example run:
>
> $ bumpversion patch --list
> current_version=(1, 11, 0, 'alpha', 1)
> new_version=(1, 11, 1, 'alpha', 0)
>
> $ bumpversion patch --list
> current_version=(1, 11, 1, 'alpha', 0)
> new_version=(1, 11, 2, 'alpha', 0)
>
> $ bumpversion minor --list
> current_version=(1, 11, 2, 'alpha', 0)
> new_version=(1, 12, 0, 'alpha', 0)
>
> $ bumpversion prerel --list
> current_version=(1, 12, 0, 'alpha', 0)
> new_version=(1, 12, 0, 'beta', 0)
>
> $ bumpversion prerel --list
> current_version=(1, 12, 0, 'beta', 0)
> new_version=(1, 12, 0, 'rc', 0)
>
> $ bumpversion prerelnum --list
> current_version=(1, 12, 0, 'rc', 0)
> new_version=(1, 12, 0, 'rc', 1)
>
> $ bumpversion major --list
> current_version=(1, 12, 0, 'rc', 1)
> new_version=(2, 0, 0, 'alpha', 0)
>
> With regards to Jenkins, you could have a release project with parameters
> of BRANCH and VERSIONPART and whatever else was needed. The VERSIONPART
> would feed into the bumpversion command.
>
> I think this way reduces backward compatibility concerns.
>
> On Saturday, 26 November 2016 06:49:54 UTC+11, Tim Graham wrote:
>>
>> After doing releases about once a month for a while, I'm thinking it
>> would be nice if releasing Django could be a bit more automated. As far as
>> I know, the process hasn't changed too much in 10 years, while the state of
>> Python packaging has improved.
>>
>> Currently doing a release requires bumping the VERSION tuple in
>> django/__init__.py [0], creating tarball/wheel and checksum files, asking
>> someone in IRC to verify the checks look fine, and upload files to various
>> locations [1]. My idea would be to use setuptools_scm [2] (that choice
>> because this is the only tool I know about) to eliminate the need to
>> bump/set a VERSION tuple and instead fetch the version information based on
>> git tags. Something like (based on what I found in djanog-hosts [3]):
>>
>> __version__ = pkg_resources.get_distribution('django').version
>>
>> One issue I'm not sure how to deal with is how to provides
>> backwards-compatibility for django.VERSION, the tuple format that looks
>> something like:
>>
>> VERSION = (1, 11, 0, 'alpha', 0)
>>
>> I think removing or deprecating it would be quite disruptive for the
>> community as I've seen lots of third-party apps that do checks like "if
>> django.VERSION < (1, 9): ...", for example. I suspect it's possible to
>> parse the version string to create something similar, but I wondered if
>> anyone has experience with that.
>>
>> Finally, uploading to PyPI and djangoproject.com (release and checksum
>> files) could be done automatically by Jenkins or something similar.
>>
>> Thanks for your comments, concerns, and suggestions.
>>
>> [0] https://github.com/django/django/blob/6252fd6314f66d2d303cc4
>> 7c791ffefd27169b42/django/__init__.py#L5
>> [1] https://docs.djangoproject.com/en/dev/internals/howto-releas
>> e-django/#actually-rolling-the-release
>> [2] https://github.com/pypa/setuptools_scm
>> [3] https://github.com/jazzband/django-hosts/blob/0eae77b31927dd
>> b0267943d6f9bcfc22f14706f2/django_hosts/__init__.py#L11
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/f36b9371-f292-47c5-a417-
> c6557e5a2557%40googlegroups.com
>