Re: Default to BigAutoField

[Adam Johnson]

I agree with Tim. I also think the rename has potential to mess with
historical migrations, or other uses of the field classes, since the output
of deconstruct() will change the class name.

On 31 August 2017 at 16:13, Tim Graham  wrote:

> Glancing at the PR, one thing I'm not sure about is renaming AutoField to
> SmallAutoField. I think that's going to cause needless additional
> complexity and confusion among people who've worked with Django a long
> time. For example, you might think that "Small" there has a similar meaning
> as SmallIntegerField. Did I miss the discussion about the benefits of the
> rename? If we agree to proceed with it, please split it out into a separate
> ticket/PR for easier review.
>
> On Sunday, August 27, 2017 at 5:31:47 AM UTC-4, Adam Johnson wrote:
>>
>> I don't think "primary key size" is something that's even within
>> consideration for 99% of django apps. Sure a few more bytes are going to be
>> wasted here, but you could argue the same that the default AutoField was
>> already too big for most models that have 100's of instances and could use
>> even a one byte primary key.
>>
>> Defaulting to BigAutoField everywhere is the simple solution that stops
>> everyone from ever worrying about their tables filling up. Additionally
>> using compressed tables helps reclaim nearly all those unused bytes, at
>> least on MySQL.
>>
>> On 18 August 2017 at 17:14, Andrew Godwin  wrote:
>>
>>>
>>>
>>> On Fri, Aug 18, 2017 at 5:43 AM, Markus Holtermann <
>>> in...@markusholtermann.eu> wrote:

 I'm don't fully agree with the approach. This essentially forces 3rd
 party package authors to make the call about the primary key field size.
 While for small to medium size projects BigAutoField is unlikely
 required and only comes with additional (storage) costs. Given that the
 migrations would need to be part of the 3rd party package there's also
 no (trivial) way for project developers to force or change to
 SmallAutoField for those packages. The same thing holds the other way
 round.

 Unfortunately, I don't have another solution at hand.


>>> This is also true of changing the primary key of third-party packages in
>>> general though - e.g. there's no way I can make everything use UUIDs even
>>> if my database would be way better at those.
>>>
>>> I don't see any other solutions that aren't settings doing
>>> spooky-action-at-a-distance to primary keys, and that's something I really
>>> don't want to see.
>>>
>>> Andrew
>>>
>>>

 On Thu, Aug 17, 2017 at 02:43:07PM -0700, Andrew Godwin wrote:

> To elaborate on the solution we eventually came up with - we default
> models
> to use a new BigAutoField that migrations will pick up on and generate
> migrations to alter columns to, but for safety reasons for those that
> don't
> read release notes, made the migration autodetector ask you if you
> want to
> make these migrations with a slowness warning.
>
> It also tells you how to preserve the old behaviour and avoid new
> migrations if you wish (manually set id = SmallAutoField)
>
> I like this approach as it means no new settings or Meta options or
> anything, has a reasonable upgrade path, and won't let people
> unwittingly
> wander into giant changes. The downside is that it does add slightly
> more
> friction to the upgrade process.
>
> Andrew
>
> On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz 
> wrote:
>
> I have opened a pull request:
>>
>> https://github.com/django/django/pull/8924
>>
>> Andrew and I came up with a good solution for migrations, together at
>> DjangoCon.
>>
>> On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:
>>
>>>
>>> On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:
>>>
>>> > However, it should also be noted that those same larger
>>> applications
>>>
>>> > are the ones that are likely to run into this problem eventually,
>>> so
>>>
>>> > perhaps forcing the migration is the best path moving forward.
>>>
>>>
>>>
>>>
>>>
>>> Existing models are the problem. Then again the database knows the
>>> truth.
>>> So with a little inspection during apps.get_models we might be able
>>> to do
>>> the right thing and even allow migrating in steps.
>>>
>>>
>>>
>>> Apps is also the place to mark an app as migrated.
>>>
>>>
>>>
>>> In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
>>> inject the right one?
>>>
>>>
>>>
>>> You asked fr thoughts, so there's my 2c stream.
>>>
>>> --
>>>
>>> Melvyn Sopacua
>>>
>>> --
>> You received this message because you are subscribed to the Google
>> Groups
>> "Django developers 

Re: Default to BigAutoField

[Tim Graham]

Glancing at the PR, one thing I'm not sure about is renaming AutoField to 
SmallAutoField. I think that's going to cause needless additional 
complexity and confusion among people who've worked with Django a long 
time. For example, you might think that "Small" there has a similar meaning 
as SmallIntegerField. Did I miss the discussion about the benefits of the 
rename? If we agree to proceed with it, please split it out into a separate 
ticket/PR for easier review.

On Sunday, August 27, 2017 at 5:31:47 AM UTC-4, Adam Johnson wrote:
>
> I don't think "primary key size" is something that's even within 
> consideration for 99% of django apps. Sure a few more bytes are going to be 
> wasted here, but you could argue the same that the default AutoField was 
> already too big for most models that have 100's of instances and could use 
> even a one byte primary key.
>
> Defaulting to BigAutoField everywhere is the simple solution that stops 
> everyone from ever worrying about their tables filling up. Additionally 
> using compressed tables helps reclaim nearly all those unused bytes, at 
> least on MySQL.
>
> On 18 August 2017 at 17:14, Andrew Godwin  > wrote:
>
>>
>>
>> On Fri, Aug 18, 2017 at 5:43 AM, Markus Holtermann <
>> in...@markusholtermann.eu > wrote:
>>>
>>> I'm don't fully agree with the approach. This essentially forces 3rd
>>> party package authors to make the call about the primary key field size.
>>> While for small to medium size projects BigAutoField is unlikely
>>> required and only comes with additional (storage) costs. Given that the
>>> migrations would need to be part of the 3rd party package there's also
>>> no (trivial) way for project developers to force or change to
>>> SmallAutoField for those packages. The same thing holds the other way
>>> round.
>>>
>>> Unfortunately, I don't have another solution at hand.
>>>
>>>
>> This is also true of changing the primary key of third-party packages in 
>> general though - e.g. there's no way I can make everything use UUIDs even 
>> if my database would be way better at those.
>>
>> I don't see any other solutions that aren't settings doing 
>> spooky-action-at-a-distance to primary keys, and that's something I really 
>> don't want to see.
>>
>> Andrew
>>  
>>
>>>
>>> On Thu, Aug 17, 2017 at 02:43:07PM -0700, Andrew Godwin wrote:
>>>
 To elaborate on the solution we eventually came up with - we default 
 models
 to use a new BigAutoField that migrations will pick up on and generate
 migrations to alter columns to, but for safety reasons for those that 
 don't
 read release notes, made the migration autodetector ask you if you want 
 to
 make these migrations with a slowness warning.

 It also tells you how to preserve the old behaviour and avoid new
 migrations if you wish (manually set id = SmallAutoField)

 I like this approach as it means no new settings or Meta options or
 anything, has a reasonable upgrade path, and won't let people 
 unwittingly
 wander into giant changes. The downside is that it does add slightly 
 more
 friction to the upgrade process.

 Andrew

 On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz  wrote:

 I have opened a pull request:
>
> https://github.com/django/django/pull/8924
>
> Andrew and I came up with a good solution for migrations, together at
> DjangoCon.
>
> On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:
>
>>
>> On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:
>>
>> > However, it should also be noted that those same larger applications
>>
>> > are the ones that are likely to run into this problem eventually, so
>>
>> > perhaps forcing the migration is the best path moving forward.
>>
>>
>>
>>
>>
>> Existing models are the problem. Then again the database knows the 
>> truth.
>> So with a little inspection during apps.get_models we might be able 
>> to do
>> the right thing and even allow migrating in steps.
>>
>>
>>
>> Apps is also the place to mark an app as migrated.
>>
>>
>>
>> In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
>> inject the right one?
>>
>>
>>
>> You asked fr thoughts, so there's my 2c stream.
>>
>> --
>>
>> Melvyn Sopacua
>>
>> --
> You received this message because you are subscribed to the Google 
> Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an
> email to django-develop...@googlegroups.com .
> To post to this group, send email to django-d...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 

Re: Default to BigAutoField

[Adam Johnson]

I don't think "primary key size" is something that's even within
consideration for 99% of django apps. Sure a few more bytes are going to be
wasted here, but you could argue the same that the default AutoField was
already too big for most models that have 100's of instances and could use
even a one byte primary key.

Defaulting to BigAutoField everywhere is the simple solution that stops
everyone from ever worrying about their tables filling up. Additionally
using compressed tables helps reclaim nearly all those unused bytes, at
least on MySQL.

On 18 August 2017 at 17:14, Andrew Godwin  wrote:

>
>
> On Fri, Aug 18, 2017 at 5:43 AM, Markus Holtermann <
> i...@markusholtermann.eu> wrote:
>>
>> I'm don't fully agree with the approach. This essentially forces 3rd
>> party package authors to make the call about the primary key field size.
>> While for small to medium size projects BigAutoField is unlikely
>> required and only comes with additional (storage) costs. Given that the
>> migrations would need to be part of the 3rd party package there's also
>> no (trivial) way for project developers to force or change to
>> SmallAutoField for those packages. The same thing holds the other way
>> round.
>>
>> Unfortunately, I don't have another solution at hand.
>>
>>
> This is also true of changing the primary key of third-party packages in
> general though - e.g. there's no way I can make everything use UUIDs even
> if my database would be way better at those.
>
> I don't see any other solutions that aren't settings doing
> spooky-action-at-a-distance to primary keys, and that's something I really
> don't want to see.
>
> Andrew
>
>
>>
>> On Thu, Aug 17, 2017 at 02:43:07PM -0700, Andrew Godwin wrote:
>>
>>> To elaborate on the solution we eventually came up with - we default
>>> models
>>> to use a new BigAutoField that migrations will pick up on and generate
>>> migrations to alter columns to, but for safety reasons for those that
>>> don't
>>> read release notes, made the migration autodetector ask you if you want
>>> to
>>> make these migrations with a slowness warning.
>>>
>>> It also tells you how to preserve the old behaviour and avoid new
>>> migrations if you wish (manually set id = SmallAutoField)
>>>
>>> I like this approach as it means no new settings or Meta options or
>>> anything, has a reasonable upgrade path, and won't let people unwittingly
>>> wander into giant changes. The downside is that it does add slightly more
>>> friction to the upgrade process.
>>>
>>> Andrew
>>>
>>> On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz 
>>> wrote:
>>>
>>> I have opened a pull request:

 https://github.com/django/django/pull/8924

 Andrew and I came up with a good solution for migrations, together at
 DjangoCon.

 On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:

>
> On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:
>
> > However, it should also be noted that those same larger applications
>
> > are the ones that are likely to run into this problem eventually, so
>
> > perhaps forcing the migration is the best path moving forward.
>
>
>
>
>
> Existing models are the problem. Then again the database knows the
> truth.
> So with a little inspection during apps.get_models we might be able to
> do
> the right thing and even allow migrating in steps.
>
>
>
> Apps is also the place to mark an app as migrated.
>
>
>
> In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
> inject the right one?
>
>
>
> You asked fr thoughts, so there's my 2c stream.
>
> --
>
> Melvyn Sopacua
>
> --
 You received this message because you are subscribed to the Google
 Groups
 "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an
 email to django-developers+unsubscr...@googlegroups.com.
 To post to this group, send email to django-developers@googlegroups.com
 .
 Visit this group at https://groups.google.com/group/django-developers.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/django-developers/e3effc41-10e1-42e2-9037-
 84c98217cd91%40googlegroups.com
 
 .

 For more options, visit https://groups.google.com/d/optout.


>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers  (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-developers+unsubscr...@googlegroups.com.
>>> To post to this group, send email to 

Re: Default to BigAutoField

[Andrew Godwin]

On Fri, Aug 18, 2017 at 5:43 AM, Markus Holtermann  wrote:
>
> I'm don't fully agree with the approach. This essentially forces 3rd
> party package authors to make the call about the primary key field size.
> While for small to medium size projects BigAutoField is unlikely
> required and only comes with additional (storage) costs. Given that the
> migrations would need to be part of the 3rd party package there's also
> no (trivial) way for project developers to force or change to
> SmallAutoField for those packages. The same thing holds the other way
> round.
>
> Unfortunately, I don't have another solution at hand.
>
>
This is also true of changing the primary key of third-party packages in
general though - e.g. there's no way I can make everything use UUIDs even
if my database would be way better at those.

I don't see any other solutions that aren't settings doing
spooky-action-at-a-distance to primary keys, and that's something I really
don't want to see.

Andrew


>
> On Thu, Aug 17, 2017 at 02:43:07PM -0700, Andrew Godwin wrote:
>
>> To elaborate on the solution we eventually came up with - we default
>> models
>> to use a new BigAutoField that migrations will pick up on and generate
>> migrations to alter columns to, but for safety reasons for those that
>> don't
>> read release notes, made the migration autodetector ask you if you want to
>> make these migrations with a slowness warning.
>>
>> It also tells you how to preserve the old behaviour and avoid new
>> migrations if you wish (manually set id = SmallAutoField)
>>
>> I like this approach as it means no new settings or Meta options or
>> anything, has a reasonable upgrade path, and won't let people unwittingly
>> wander into giant changes. The downside is that it does add slightly more
>> friction to the upgrade process.
>>
>> Andrew
>>
>> On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz 
>> wrote:
>>
>> I have opened a pull request:
>>>
>>> https://github.com/django/django/pull/8924
>>>
>>> Andrew and I came up with a good solution for migrations, together at
>>> DjangoCon.
>>>
>>> On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:
>>>

 On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:

 > However, it should also be noted that those same larger applications

 > are the ones that are likely to run into this problem eventually, so

 > perhaps forcing the migration is the best path moving forward.





 Existing models are the problem. Then again the database knows the
 truth.
 So with a little inspection during apps.get_models we might be able to
 do
 the right thing and even allow migrating in steps.



 Apps is also the place to mark an app as migrated.



 In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
 inject the right one?



 You asked fr thoughts, so there's my 2c stream.

 --

 Melvyn Sopacua

 --
>>> You received this message because you are subscribed to the Google Groups
>>> "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to django-developers+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-developers@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/django-developers/e3effc41-10e1-42e2-9037-
>>> 84c98217cd91%40googlegroups.com
>>> >> 1-10e1-42e2-9037-84c98217cd91%40googlegroups.com?utm_medium=
>>> email_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/django-developers/CAFwN1uo4Y_pWSf3zAe_4R0GGkDqBv1YGus8Q%
>> 2BWPPZ%3DZ6FPwdYQ%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at 

Re: Default to BigAutoField

[Markus Holtermann]

Thanks for taking the effort to work on this, Kenneth!

I'm don't fully agree with the approach. This essentially forces 3rd
party package authors to make the call about the primary key field size.
While for small to medium size projects BigAutoField is unlikely
required and only comes with additional (storage) costs. Given that the
migrations would need to be part of the 3rd party package there's also
no (trivial) way for project developers to force or change to
SmallAutoField for those packages. The same thing holds the other way
round.

Unfortunately, I don't have another solution at hand.

I realized that I'm a bit late to the discussion and should've chimed
in before all that work was done. Please accept my apologies for that.

/Markus


On Thu, Aug 17, 2017 at 02:43:07PM -0700, Andrew Godwin wrote:

To elaborate on the solution we eventually came up with - we default models
to use a new BigAutoField that migrations will pick up on and generate
migrations to alter columns to, but for safety reasons for those that don't
read release notes, made the migration autodetector ask you if you want to
make these migrations with a slowness warning.

It also tells you how to preserve the old behaviour and avoid new
migrations if you wish (manually set id = SmallAutoField)

I like this approach as it means no new settings or Meta options or
anything, has a reasonable upgrade path, and won't let people unwittingly
wander into giant changes. The downside is that it does add slightly more
friction to the upgrade process.

Andrew

On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz  wrote:


I have opened a pull request:

https://github.com/django/django/pull/8924

Andrew and I came up with a good solution for migrations, together at
DjangoCon.

On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:


On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:

> However, it should also be noted that those same larger applications

> are the ones that are likely to run into this problem eventually, so

> perhaps forcing the migration is the best path moving forward.





Existing models are the problem. Then again the database knows the truth.
So with a little inspection during apps.get_models we might be able to do
the right thing and even allow migrating in steps.



Apps is also the place to mark an app as migrated.



In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
inject the right one?



You asked fr thoughts, so there's my 2c stream.

--

Melvyn Sopacua


--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/e3effc41-10e1-42e2-9037-
84c98217cd91%40googlegroups.com

.

For more options, visit https://groups.google.com/d/optout.



--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uo4Y_pWSf3zAe_4R0GGkDqBv1YGus8Q%2BWPPZ%3DZ6FPwdYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20170818124354.GA1898%40inel.local.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Andrew Godwin]

To elaborate on the solution we eventually came up with - we default models
to use a new BigAutoField that migrations will pick up on and generate
migrations to alter columns to, but for safety reasons for those that don't
read release notes, made the migration autodetector ask you if you want to
make these migrations with a slowness warning.

It also tells you how to preserve the old behaviour and avoid new
migrations if you wish (manually set id = SmallAutoField)

I like this approach as it means no new settings or Meta options or
anything, has a reasonable upgrade path, and won't let people unwittingly
wander into giant changes. The downside is that it does add slightly more
friction to the upgrade process.

Andrew

On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz  wrote:

> I have opened a pull request:
>
> https://github.com/django/django/pull/8924
>
> Andrew and I came up with a good solution for migrations, together at
> DjangoCon.
>
> On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:
>>
>> On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:
>>
>> > However, it should also be noted that those same larger applications
>>
>> > are the ones that are likely to run into this problem eventually, so
>>
>> > perhaps forcing the migration is the best path moving forward.
>>
>>
>>
>>
>>
>> Existing models are the problem. Then again the database knows the truth.
>> So with a little inspection during apps.get_models we might be able to do
>> the right thing and even allow migrating in steps.
>>
>>
>>
>> Apps is also the place to mark an app as migrated.
>>
>>
>>
>> In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
>> inject the right one?
>>
>>
>>
>> You asked fr thoughts, so there's my 2c stream.
>>
>> --
>>
>> Melvyn Sopacua
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/e3effc41-10e1-42e2-9037-
> 84c98217cd91%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uo4Y_pWSf3zAe_4R0GGkDqBv1YGus8Q%2BWPPZ%3DZ6FPwdYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Melvyn Sopacua]

On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:
> However, it should also be noted that those same larger applications
> are the ones that are likely to run into this problem eventually, so
> perhaps forcing the migration is the best path moving forward.


Existing models are the problem. Then again the database knows the truth. So 
with a little inspection during apps.get_models we might be able to do the 
right 
thing and even allow migrating in steps.

Apps is also the place to mark an app as migrated.

In fact - couldn't an AppConfig grow a method "get_autoid_type()" and inject 
the right one?

You asked fr thoughts, so there's my 2c stream.
-- 
Melvyn Sopacua

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8118805.cf4lglgPtt%40devstation.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Ash Christopher]

I really like this idea. We created a custom BigAutoField for our systems 
with sharded databases, and fast growing data.

I understand the desire to fix it for new projects moving forward, but this 
smells a little like what happened with custom User models - it was 
introduced for new projects, but there was no clear upgrade/migration path 
for existing projects. 


On Friday, 9 June 2017 15:54:43 UTC-4, Kenneth Reitz wrote:
>
> My initial thought was to just have this apply to *new* Django 
> applications, if that's possible.
>
> Running this migration could take quite some time on some larger apps, and 
> would require a non-trivial amount of time to execute. 
>
> --
> Kenneth Reitz
>
> On Jun 9, 2017, at 3:53 PM, Tom Forbes  
> wrote:
>
> How would this work with generic foreign keys as well? Those migrations 
> couldn't be automatic.
>
> On 9 Jun 2017 20:43, "Jacob Kaplan-Moss"  
> wrote:
>
>> I think this would be a good improvement, and I'd like to see it. I've 
>> been bitten by integers overflowing at least twice I can remember in my 
>> career, which is two times too many.
>>
>> However, a major thing we'd have to work out is the upgrade path Consider 
>> a simple model:
>>
>> class Person(Model):
>> name = CharField()
>>
>> In Django 1.11, this actually generates a model with an integer `id` 
>> field. But in we change it, in Django vNext, that `id` field would "turn 
>> into" a bigint magically without the underlying table changes. That'd be 
>> confusing: you'd expect the model to be "fixed" by pugrading to vNext, but 
>> it wouldn't be. I think the migrations engine would detect this as a 
>> migration (?), so perhaps that's the path forward, but it could still be 
>> super-confusing. We've never shipped a release of Django that required a 
>> migration to _all_ your models.
>>
>> Have you thought about what the upgrade path should look like, Kenneth?
>>
>> Jacob
>>
>> On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz > > wrote:
>>
>>> Dear Django Dev,
>>>
>>>  
>>> At Heroku, we have the privilege of seeing an extremely broad range of 
>>> customers utilizing tools like Django to build their applications and 
>>> companies. One of the things that we’ve seen customers hit, time and time 
>>> again when using tools like Django, is integer overflows for primary keys. 
>>> Their application starts behaving unpredictably once they reach the 
>>> overflow, not even knowing such a constraint exists, and they often think 
>>> the problem is with their database provider, rather than with their schema. 
>>> Once they realize what is wrong, it’s a relatively trivial fix, but a 
>>> migration can take several hours to complete, which results in unacceptable 
>>> amounts of downtime.
>>>
>>>  
>>> Because of this, Heroku, as a company, would like to encourage bigints 
>>> as a sane reasonable default for primary keys for application models. If 
>>> the Django project agrees with this idea, that would mean that Django would 
>>> provide BigAutoField as the default for ‘id’ instead of AutoField. 
>>>
>>>  
>>> Rails made this change recently 
>>> , and has seen 
>>> success in doing so. 
>>>
>>>  
>>> I’m happy to provide the code to do this, but I wanted to discuss it 
>>> here before doing so, to hear what the general consensus was to the 
>>> proposal of such a change. 
>>>
>>>  
>>>
>>>  
>>> Pros:
>>>
>>>- Users of Django, following the docs, won’t accidentally hit the 
>>>int overflow barrier. 
>>>- Encourages best-practices from the beginning. 
>>>- Bigints don’t take up much more storage than ints when using 
>>>Postgres. 
>>>- In-line with other frameworks moving forward on this issue, like 
>>>Rails .
>>>
>>>  
>>> Cons:
>>>
>>>- Backwards compatibility would need to be considered. 
>>>
>>>  
>>> Why not UUID?
>>>
>>>  
>>> I agree! I love using UUID for my primary keys, and I think a patch to 
>>> Django which provides an AutoUUIDField would be wonderful. However, there 
>>> are a few major drawbacks to making this the new default:
>>>
>>>  
>>>
>>>1. It’s confusing to new users, would make onboarding process more 
>>>difficult. 
>>>2. UUID is difficult to implement in MySQL. 
>>>3. UUID has larger storage requirements. 
>>>4. Incrementing IDs are actually useful. 
>>>
>>>  
>>>
>>> So, my proposal is to simply lift the int barrier to a bigint barrier 
>>> for new Django applications, and I think it will save a lot of developers a 
>>> lot of pain in the long run. 
>>>
>>>  
>>> Many thanks,
>>>
>>>  
>>> Kenneth Reitz
>>> Heroku Python
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, 

Re: Default to BigAutoField

[emorley]

On Saturday, 10 June 2017 10:33:35 UTC+1, Claude Paroz wrote:
>
> Another idea is to leverage the system check framework (--deploy part) to 
> warn when the max id is over 50% of available range.
>

I was about to suggest the same. Seems like something worth doing 
regardless of whether we change the default (since it would help catch 
issues in existing installations who haven't yet migrated).

Ed

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fe0c7429-e9e9-406f-8701-8ebc550641e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Claude Paroz]

Le samedi 10 juin 2017 14:25:49 UTC+2, Curtis Maloney a écrit :

> On 10/06/17 22:21, Claude Paroz wrote: 
> > Another idea would be to offer variants of models.Model which models 
> > could inherit from, like models.BigIntModel or models.UUIDModel. 
>
> Ah, well... now you're talking.  But then, you can do this already as an 
> abstract base with TYPE as id... 
>
> class BigIntModel(models.Model): 
>  id = BigAutoField(primary_key=True) 
>  class Meta: 
>  abstract = True 
>

Absolutely! It would only be a convenience thing. The more I think about 
this, the more I think that it might be addressed by better documentation 
(and maybe some convenience shortcuts). 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d6cbd561-190e-40b5-8419-aac3e42f66b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Curtis Maloney]

On 10/06/17 22:21, Claude Paroz wrote:

Le samedi 10 juin 2017 11:40:42 UTC+2, Curtis Maloney a écrit :

Right, hence my point of having a global setting to say "the default
auto-field is ..."



I see, but this conforms to the pattern "use the same id field type for
all models of my project". I'm not sure we should encourage that.


Yeah... naming would be key with "DEFAULT_AUTO_FIELD_TYPE" or whatever...


Another idea would be to offer variants of models.Model which models
could inherit from, like models.BigIntModel or models.UUIDModel.


Ah, well... now you're talking.  But then, you can do this already as an 
abstract base with TYPE as id...


class BigIntModel(models.Model):
id = BigAutoField(primary_key=True)
class Meta:
abstract = True


--
C

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a7da67fb-349a-c77c-8324-32dc16bd0ada%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Claude Paroz]

Le samedi 10 juin 2017 11:40:42 UTC+2, Curtis Maloney a écrit :
>
> Right, hence my point of having a global setting to say "the default 
> auto-field is ..." 
>
> This would: 
> a) ...
>

I see, but this conforms to the pattern "use the same id field type for all 
models of my project". I'm not sure we should encourage that.

Another idea would be to offer variants of models.Model which models could 
inherit from, like models.BigIntModel or models.UUIDModel.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3904d3f9-2ed8-432a-83c3-3bbac1dacc55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Tom Forbes]

I really like the idea of a global configurable setting. One problem with a
setting is that it's not always changeable, which settings kind of imply
(IMO). Going from int->bigint is always possible, but going backwards may
not be, nor might going from int->guid.

I attempted an int->guid migration on one of our systems and ran into some
migrations and postgres-specific issues. Seems like it might be incredibly
difficult to get a simple toggle setting to work with it, and there are
lots of pitfalls.

A warning to the system check framework could be added to check if the
primary key type is changed and potentially incompatible generic foreign
key columns exist. And in the docs rather than use a plain IntegerField,
just use a field that changes based on the setting.

On 10 Jun 2017 10:41, "Curtis Maloney"  wrote:

f) let MySQL users opt for PositiveBigAutoField if they want...



On 10/06/17 19:40, Curtis Maloney wrote:

> Right, hence my point of having a global setting to say "the default
> auto-field is ..."
>
> This would:
> a) let people continue to use AutoField
> b) let people opt for BigAutoField
> c) let 3rd party projects be agnostic
> d) let people use UUIDField(default=uuid.uuid4)
> e) possibly make it feasible to change mid-stream from AutoField to
> BigAutoField...
>
> --
> C
>
> On 10/06/17 19:33, Claude Paroz wrote:
>
>> I think we should first discuss if it makes sense to set a BigAutoField
>> by default for all tables. I'm not convinced of that yet. I looked at my
>> projects with this perspective and found for example a case where I have
>> many small lookup tables (containing between 2-20 entries) for which I
>> know I would never use BigAutoField if I'd design the schema on paper.
>>
>> For me, it's a bit like the `on_delete` parameter for foreign keys. A no
>> "one size fits all" situation.
>> For example, a quick analysis of contrib models (sorry for bad
>> formatting):
>>
>> Model BigAutoField appropriate
>> ===
>> admin.LogEntry yes
>> auth.Userno
>> auth.Group  no
>> auth.Permission no
>> contenttype.ContentType no
>> flatpage.FlatPage  no
>> redirect.Redirectno
>> sessions.Session  yes
>> sites.Site no
>>
>> Shouldn't we treat that issue by better documentation instead?
>> Another idea is to leverage the system check framework (--deploy part)
>> to warn when the max id is over 50% of available range.
>> We are perfectionists, aren't we :-)
>>
>> Claude
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to django-developers+unsubscr...@googlegroups.com
>> .
>> To post to this group, send email to django-developers@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/0fba4c2c
>> -cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com
>>
>> > c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com?utm_medium=
>> email_source=footer>.
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/ms
gid/django-developers/ec59bb2e-bce9-924f-1509-60709a07c5dc%40tinbrain.net.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMntP93kaJ%2BbPuHzAr2rKqZbbSbu1TRc%2Bm%2Bp-%2BiO%2BojvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Curtis Maloney]

f) let MySQL users opt for PositiveBigAutoField if they want...


On 10/06/17 19:40, Curtis Maloney wrote:

Right, hence my point of having a global setting to say "the default
auto-field is ..."

This would:
a) let people continue to use AutoField
b) let people opt for BigAutoField
c) let 3rd party projects be agnostic
d) let people use UUIDField(default=uuid.uuid4)
e) possibly make it feasible to change mid-stream from AutoField to
BigAutoField...

--
C

On 10/06/17 19:33, Claude Paroz wrote:

I think we should first discuss if it makes sense to set a BigAutoField
by default for all tables. I'm not convinced of that yet. I looked at my
projects with this perspective and found for example a case where I have
many small lookup tables (containing between 2-20 entries) for which I
know I would never use BigAutoField if I'd design the schema on paper.

For me, it's a bit like the `on_delete` parameter for foreign keys. A no
"one size fits all" situation.
For example, a quick analysis of contrib models (sorry for bad
formatting):

Model BigAutoField appropriate
===
admin.LogEntry yes
auth.Userno
auth.Group  no
auth.Permission no
contenttype.ContentType no
flatpage.FlatPage  no
redirect.Redirectno
sessions.Session  yes
sites.Site no

Shouldn't we treat that issue by better documentation instead?
Another idea is to leverage the system check framework (--deploy part)
to warn when the max id is over 50% of available range.
We are perfectionists, aren't we :-)

Claude

--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to django-developers+unsubscr...@googlegroups.com
.
To post to this group, send email to django-developers@googlegroups.com
.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com

.

For more options, visit https://groups.google.com/d/optout.




--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ec59bb2e-bce9-924f-1509-60709a07c5dc%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Curtis Maloney]

Right, hence my point of having a global setting to say "the default 
auto-field is ..."


This would:
a) let people continue to use AutoField
b) let people opt for BigAutoField
c) let 3rd party projects be agnostic
d) let people use UUIDField(default=uuid.uuid4)
e) possibly make it feasible to change mid-stream from AutoField to 
BigAutoField...


--
C

On 10/06/17 19:33, Claude Paroz wrote:

I think we should first discuss if it makes sense to set a BigAutoField
by default for all tables. I'm not convinced of that yet. I looked at my
projects with this perspective and found for example a case where I have
many small lookup tables (containing between 2-20 entries) for which I
know I would never use BigAutoField if I'd design the schema on paper.

For me, it's a bit like the `on_delete` parameter for foreign keys. A no
"one size fits all" situation.
For example, a quick analysis of contrib models (sorry for bad formatting):

Model BigAutoField appropriate
===
admin.LogEntry yes
auth.Userno
auth.Group  no
auth.Permission no
contenttype.ContentType no
flatpage.FlatPage  no
redirect.Redirectno
sessions.Session  yes
sites.Site no

Shouldn't we treat that issue by better documentation instead?
Another idea is to leverage the system check framework (--deploy part)
to warn when the max id is over 50% of available range.
We are perfectionists, aren't we :-)

Claude

--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to django-developers+unsubscr...@googlegroups.com
.
To post to this group, send email to django-developers@googlegroups.com
.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/43c2a63f-87de-fda1-87c3-412574c13804%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Claude Paroz]

I think we should first discuss if it makes sense to set a BigAutoField by 
default for all tables. I'm not convinced of that yet. I looked at my 
projects with this perspective and found for example a case where I have 
many small lookup tables (containing between 2-20 entries) for which I know 
I would never use BigAutoField if I'd design the schema on paper.

For me, it's a bit like the `on_delete` parameter for foreign keys. A no 
"one size fits all" situation.
For example, a quick analysis of contrib models (sorry for bad formatting):

Model BigAutoField appropriate
===
admin.LogEntry yes
auth.Userno
auth.Group  no
auth.Permission no
contenttype.ContentType no
flatpage.FlatPage  no
redirect.Redirectno
sessions.Session  yes
sites.Site no

Shouldn't we treat that issue by better documentation instead?
Another idea is to leverage the system check framework (--deploy part) to 
warn when the max id is over 50% of available range.
We are perfectionists, aren't we :-)

Claude

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Andrew Godwin]

As long as you changed the actual field that ended up under the "id"
column, then yes, migrations should detect it and apply all the changes
during an upgrade, including to foreign keys.

Generic foreign keys are another problem, however, not to mention the
prospect of how many migrations and ALTER TABLES this would result in on a
large project. We could include in the upgrade notes how to stave it off
(explicitly specify an ID field), but I'm sort of on the fence as to if
that's good enough.

Andrew

On Sat, Jun 10, 2017 at 9:53 AM, Collin Anderson 
wrote:

> I might be wrong, but if the default changes, won't the migrations detect
> it and migrate it just fine, including foreign keys?
>
> All of my migrations have this:
> ('id', models.AutoField(auto_created=True, primary_key=True,
> serialize=False, verbose_name='ID')),
>
> So everyone would either need to manually specify the AutoField to keep
> the old behavior, or run makemigrations to auto-generate migrations to
> BigAutoField. This seems similar to increasing the max_length of
> EmailField, user.username, and user.last_name, though would affect a lot
> more models in this case.
>
> (I'm not sure what I think about the setting idea.)
>
> (While we're at it, maybe we could make it a (new) PositiveBigAutoField to
> help out the mysql folks and close the oldest open ticket:
> https://code.djangoproject.com/ticket/56 :)
>
>
>
> On Fri, Jun 9, 2017 at 9:37 PM, Tim Graham  wrote:
>
>> I'm not sure how this could work with migrations. In a sense, it would
>> involve making the auto-generated primary key "swappable", including
>> foreign keys that point to it. This sounds like a headache.
>>
>> I haven't thought of a possible solution since Kenneth floated this idea
>> in #django-dev yesterday.
>>
>> On Friday, June 9, 2017 at 7:11:05 PM UTC-4, Curtis Maloney wrote:
>>>
>>> I know people hate settings, but what about one for auto id field type?
>>>
>>> It would let you handle backwards compatibility, uuid, and bigint...
>>>
>>> --
>>> C
>>>
>>>
>>> On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss 
>>> wrote:

 I think this would be a good improvement, and I'd like to see it. I've
 been bitten by integers overflowing at least twice I can remember in my
 career, which is two times too many.

 However, a major thing we'd have to work out is the upgrade path
 Consider a simple model:

 class Person(Model):
 name = CharField()

 In Django 1.11, this actually generates a model with an integer `id`
 field. But in we change it, in Django vNext, that `id` field would "turn
 into" a bigint magically without the underlying table changes. That'd be
 confusing: you'd expect the model to be "fixed" by pugrading to vNext, but
 it wouldn't be. I think the migrations engine would detect this as a
 migration (?), so perhaps that's the path forward, but it could still be
 super-confusing. We've never shipped a release of Django that required a
 migration to _all_ your models.

 Have you thought about what the upgrade path should look like, Kenneth?

 Jacob

 On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz 
 wrote:

> Dear Django Dev,
>
>
>
> At Heroku, we have the privilege of seeing an extremely broad range of
> customers utilizing tools like Django to build their applications and
> companies. One of the things that we’ve seen customers hit, time and time
> again when using tools like Django, is integer overflows for primary keys.
> Their application starts behaving unpredictably once they reach the
> overflow, not even knowing such a constraint exists, and they often think
> the problem is with their database provider, rather than with their 
> schema.
> Once they realize what is wrong, it’s a relatively trivial fix, but a
> migration can take several hours to complete, which results in 
> unacceptable
> amounts of downtime.
>
>
>
> Because of this, Heroku, as a company, would like to encourage bigints
> as a sane reasonable default for primary keys for application models. If
> the Django project agrees with this idea, that would mean that Django 
> would
> provide BigAutoField as the default for ‘id’ instead of AutoField.
>
>
>
> Rails made this change recently
> , and has seen
> success in doing so.
>
>
>
> I’m happy to provide the code to do this, but I wanted to discuss it
> here before doing so, to hear what the general consensus was to the
> proposal of such a change.
>
>
>
>
>
> Pros:
>
>-
>
>Users of Django, following the docs, won’t accidentally hit the
>int overflow barrier.
>-
>
>Encourages 

Re: Default to BigAutoField

[Collin Anderson]

I might be wrong, but if the default changes, won't the migrations detect
it and migrate it just fine, including foreign keys?

All of my migrations have this:
('id', models.AutoField(auto_created=True, primary_key=True,
serialize=False, verbose_name='ID')),

So everyone would either need to manually specify the AutoField to keep the
old behavior, or run makemigrations to auto-generate migrations to
BigAutoField. This seems similar to increasing the max_length of
EmailField, user.username, and user.last_name, though would affect a lot
more models in this case.

(I'm not sure what I think about the setting idea.)

(While we're at it, maybe we could make it a (new) PositiveBigAutoField to
help out the mysql folks and close the oldest open ticket:
https://code.djangoproject.com/ticket/56 :)



On Fri, Jun 9, 2017 at 9:37 PM, Tim Graham  wrote:

> I'm not sure how this could work with migrations. In a sense, it would
> involve making the auto-generated primary key "swappable", including
> foreign keys that point to it. This sounds like a headache.
>
> I haven't thought of a possible solution since Kenneth floated this idea
> in #django-dev yesterday.
>
> On Friday, June 9, 2017 at 7:11:05 PM UTC-4, Curtis Maloney wrote:
>>
>> I know people hate settings, but what about one for auto id field type?
>>
>> It would let you handle backwards compatibility, uuid, and bigint...
>>
>> --
>> C
>>
>>
>> On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss 
>> wrote:
>>>
>>> I think this would be a good improvement, and I'd like to see it. I've
>>> been bitten by integers overflowing at least twice I can remember in my
>>> career, which is two times too many.
>>>
>>> However, a major thing we'd have to work out is the upgrade path
>>> Consider a simple model:
>>>
>>> class Person(Model):
>>> name = CharField()
>>>
>>> In Django 1.11, this actually generates a model with an integer `id`
>>> field. But in we change it, in Django vNext, that `id` field would "turn
>>> into" a bigint magically without the underlying table changes. That'd be
>>> confusing: you'd expect the model to be "fixed" by pugrading to vNext, but
>>> it wouldn't be. I think the migrations engine would detect this as a
>>> migration (?), so perhaps that's the path forward, but it could still be
>>> super-confusing. We've never shipped a release of Django that required a
>>> migration to _all_ your models.
>>>
>>> Have you thought about what the upgrade path should look like, Kenneth?
>>>
>>> Jacob
>>>
>>> On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz 
>>> wrote:
>>>
 Dear Django Dev,



 At Heroku, we have the privilege of seeing an extremely broad range of
 customers utilizing tools like Django to build their applications and
 companies. One of the things that we’ve seen customers hit, time and time
 again when using tools like Django, is integer overflows for primary keys.
 Their application starts behaving unpredictably once they reach the
 overflow, not even knowing such a constraint exists, and they often think
 the problem is with their database provider, rather than with their schema.
 Once they realize what is wrong, it’s a relatively trivial fix, but a
 migration can take several hours to complete, which results in unacceptable
 amounts of downtime.



 Because of this, Heroku, as a company, would like to encourage bigints
 as a sane reasonable default for primary keys for application models. If
 the Django project agrees with this idea, that would mean that Django would
 provide BigAutoField as the default for ‘id’ instead of AutoField.



 Rails made this change recently
 , and has seen
 success in doing so.



 I’m happy to provide the code to do this, but I wanted to discuss it
 here before doing so, to hear what the general consensus was to the
 proposal of such a change.





 Pros:

-

Users of Django, following the docs, won’t accidentally hit the int
overflow barrier.
-

Encourages best-practices from the beginning.
-

Bigints don’t take up much more storage than ints when using
Postgres.
-

In-line with other frameworks moving forward on this issue, like
Rails .



 Cons:

-

Backwards compatibility would need to be considered.


 Why not UUID?



 I agree! I love using UUID for my primary keys, and I think a patch to
 Django which provides an AutoUUIDField would be wonderful. However, there
 are a few major drawbacks to making this the new default:



1.

It’s confusing to new users, would make 

Re: Default to BigAutoField

[Curtis Maloney]

I don't see it helping once you hit the problem, but like custom user its a 
recommendable setting for pre planning

--
C


On 10 June 2017 11:37:04 AM AEST, Tim Graham  wrote:
>I'm not sure how this could work with migrations. In a sense, it would 
>involve making the auto-generated primary key "swappable", including 
>foreign keys that point to it. This sounds like a headache.
>
>I haven't thought of a possible solution since Kenneth floated this
>idea in 
>#django-dev yesterday.
>
>On Friday, June 9, 2017 at 7:11:05 PM UTC-4, Curtis Maloney wrote:
>>
>> I know people hate settings, but what about one for auto id field
>type?
>>
>> It would let you handle backwards compatibility, uuid, and bigint...
>>
>> --
>> C
>>
>>
>> On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss
>> > wrote:
>>>
>>> I think this would be a good improvement, and I'd like to see it.
>I've 
>>> been bitten by integers overflowing at least twice I can remember in
>my 
>>> career, which is two times too many.
>>>
>>> However, a major thing we'd have to work out is the upgrade path
>Consider 
>>> a simple model:
>>>
>>> class Person(Model):
>>> name = CharField()
>>>
>>> In Django 1.11, this actually generates a model with an integer `id`
>
>>> field. But in we change it, in Django vNext, that `id` field would
>"turn 
>>> into" a bigint magically without the underlying table changes.
>That'd be 
>>> confusing: you'd expect the model to be "fixed" by pugrading to
>vNext, but 
>>> it wouldn't be. I think the migrations engine would detect this as a
>
>>> migration (?), so perhaps that's the path forward, but it could
>still be 
>>> super-confusing. We've never shipped a release of Django that
>required a 
>>> migration to _all_ your models.
>>>
>>> Have you thought about what the upgrade path should look like,
>Kenneth?
>>>
>>> Jacob
>>>
>>> On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz >> > wrote:
>>>
 Dear Django Dev,

  

 At Heroku, we have the privilege of seeing an extremely broad range
>of 
 customers utilizing tools like Django to build their applications
>and 
 companies. One of the things that we’ve seen customers hit, time
>and time 
 again when using tools like Django, is integer overflows for
>primary keys. 
 Their application starts behaving unpredictably once they reach the
>
 overflow, not even knowing such a constraint exists, and they often
>think 
 the problem is with their database provider, rather than with their
>schema. 
 Once they realize what is wrong, it’s a relatively trivial fix, but
>a 
 migration can take several hours to complete, which results in
>unacceptable 
 amounts of downtime.

  

 Because of this, Heroku, as a company, would like to encourage
>bigints 
 as a sane reasonable default for primary keys for application
>models. If 
 the Django project agrees with this idea, that would mean that
>Django would 
 provide BigAutoField as the default for ‘id’ instead of AutoField. 

  

 Rails made this change recently 
 , and has seen 
 success in doing so. 

  

 I’m happy to provide the code to do this, but I wanted to discuss
>it 
 here before doing so, to hear what the general consensus was to the
>
 proposal of such a change. 

  

  

 Pros:

- 

Users of Django, following the docs, won’t accidentally hit the
>int 
overflow barrier. 
- 

Encourages best-practices from the beginning. 
- 

Bigints don’t take up much more storage than ints when using 
Postgres. 
- 

In-line with other frameworks moving forward on this issue, like
>
Rails .

  

 Cons:

- 

Backwards compatibility would need to be considered. 

  
 Why not UUID?

  

 I agree! I love using UUID for my primary keys, and I think a patch
>to 
 Django which provides an AutoUUIDField would be wonderful. However,
>there 
 are a few major drawbacks to making this the new default:

  

1. 

It’s confusing to new users, would make onboarding process more 
difficult. 
2. 

UUID is difficult to implement in MySQL. 
3. 

UUID has larger storage requirements. 
4. 

Incrementing IDs are actually useful. 

  


 So, my proposal is to simply lift the int barrier to a bigint
>barrier 
 for new Django applications, and I think it will save a lot of
>developers a 
 lot of pain in the long run. 

  

 Many thanks,

  

 Kenneth Reitz

Re: Default to BigAutoField

[Tim Graham]

I'm not sure how this could work with migrations. In a sense, it would 
involve making the auto-generated primary key "swappable", including 
foreign keys that point to it. This sounds like a headache.

I haven't thought of a possible solution since Kenneth floated this idea in 
#django-dev yesterday.

On Friday, June 9, 2017 at 7:11:05 PM UTC-4, Curtis Maloney wrote:
>
> I know people hate settings, but what about one for auto id field type?
>
> It would let you handle backwards compatibility, uuid, and bigint...
>
> --
> C
>
>
> On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss  > wrote:
>>
>> I think this would be a good improvement, and I'd like to see it. I've 
>> been bitten by integers overflowing at least twice I can remember in my 
>> career, which is two times too many.
>>
>> However, a major thing we'd have to work out is the upgrade path Consider 
>> a simple model:
>>
>> class Person(Model):
>> name = CharField()
>>
>> In Django 1.11, this actually generates a model with an integer `id` 
>> field. But in we change it, in Django vNext, that `id` field would "turn 
>> into" a bigint magically without the underlying table changes. That'd be 
>> confusing: you'd expect the model to be "fixed" by pugrading to vNext, but 
>> it wouldn't be. I think the migrations engine would detect this as a 
>> migration (?), so perhaps that's the path forward, but it could still be 
>> super-confusing. We've never shipped a release of Django that required a 
>> migration to _all_ your models.
>>
>> Have you thought about what the upgrade path should look like, Kenneth?
>>
>> Jacob
>>
>> On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz > > wrote:
>>
>>> Dear Django Dev,
>>>
>>>  
>>>
>>> At Heroku, we have the privilege of seeing an extremely broad range of 
>>> customers utilizing tools like Django to build their applications and 
>>> companies. One of the things that we’ve seen customers hit, time and time 
>>> again when using tools like Django, is integer overflows for primary keys. 
>>> Their application starts behaving unpredictably once they reach the 
>>> overflow, not even knowing such a constraint exists, and they often think 
>>> the problem is with their database provider, rather than with their schema. 
>>> Once they realize what is wrong, it’s a relatively trivial fix, but a 
>>> migration can take several hours to complete, which results in unacceptable 
>>> amounts of downtime.
>>>
>>>  
>>>
>>> Because of this, Heroku, as a company, would like to encourage bigints 
>>> as a sane reasonable default for primary keys for application models. If 
>>> the Django project agrees with this idea, that would mean that Django would 
>>> provide BigAutoField as the default for ‘id’ instead of AutoField. 
>>>
>>>  
>>>
>>> Rails made this change recently 
>>> , and has seen 
>>> success in doing so. 
>>>
>>>  
>>>
>>> I’m happy to provide the code to do this, but I wanted to discuss it 
>>> here before doing so, to hear what the general consensus was to the 
>>> proposal of such a change. 
>>>
>>>  
>>>
>>>  
>>>
>>> Pros:
>>>
>>>- 
>>>
>>>Users of Django, following the docs, won’t accidentally hit the int 
>>>overflow barrier. 
>>>- 
>>>
>>>Encourages best-practices from the beginning. 
>>>- 
>>>
>>>Bigints don’t take up much more storage than ints when using 
>>>Postgres. 
>>>- 
>>>
>>>In-line with other frameworks moving forward on this issue, like 
>>>Rails .
>>>
>>>  
>>>
>>> Cons:
>>>
>>>- 
>>>
>>>Backwards compatibility would need to be considered. 
>>>
>>>  
>>> Why not UUID?
>>>
>>>  
>>>
>>> I agree! I love using UUID for my primary keys, and I think a patch to 
>>> Django which provides an AutoUUIDField would be wonderful. However, there 
>>> are a few major drawbacks to making this the new default:
>>>
>>>  
>>>
>>>1. 
>>>
>>>It’s confusing to new users, would make onboarding process more 
>>>difficult. 
>>>2. 
>>>
>>>UUID is difficult to implement in MySQL. 
>>>3. 
>>>
>>>UUID has larger storage requirements. 
>>>4. 
>>>
>>>Incrementing IDs are actually useful. 
>>>
>>>  
>>>
>>>
>>> So, my proposal is to simply lift the int barrier to a bigint barrier 
>>> for new Django applications, and I think it will save a lot of developers a 
>>> lot of pain in the long run. 
>>>
>>>  
>>>
>>> Many thanks,
>>>
>>>  
>>>
>>> Kenneth Reitz
>>>
>>> Heroku Python
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com .
>>> To post to this group, send email to django-d...@googlegroups.com 
>>> .
>>> 

Re: Default to BigAutoField

[Curtis Maloney]

I know people hate settings, but what about one for auto id field type?

It would let you handle backwards compatibility, uuid, and bigint...

--
C


On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss  wrote:
>I think this would be a good improvement, and I'd like to see it. I've
>been
>bitten by integers overflowing at least twice I can remember in my
>career,
>which is two times too many.
>
>However, a major thing we'd have to work out is the upgrade path
>Consider a
>simple model:
>
>class Person(Model):
>name = CharField()
>
>In Django 1.11, this actually generates a model with an integer `id`
>field.
>But in we change it, in Django vNext, that `id` field would "turn into"
>a
>bigint magically without the underlying table changes. That'd be
>confusing:
>you'd expect the model to be "fixed" by pugrading to vNext, but it
>wouldn't
>be. I think the migrations engine would detect this as a migration (?),
>so
>perhaps that's the path forward, but it could still be super-confusing.
>We've never shipped a release of Django that required a migration to
>_all_
>your models.
>
>Have you thought about what the upgrade path should look like, Kenneth?
>
>Jacob
>
>On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz 
>wrote:
>
>> Dear Django Dev,
>>
>>
>>
>> At Heroku, we have the privilege of seeing an extremely broad range
>of
>> customers utilizing tools like Django to build their applications and
>> companies. One of the things that we’ve seen customers hit, time and
>time
>> again when using tools like Django, is integer overflows for primary
>keys.
>> Their application starts behaving unpredictably once they reach the
>> overflow, not even knowing such a constraint exists, and they often
>think
>> the problem is with their database provider, rather than with their
>schema.
>> Once they realize what is wrong, it’s a relatively trivial fix, but a
>> migration can take several hours to complete, which results in
>unacceptable
>> amounts of downtime.
>>
>>
>>
>> Because of this, Heroku, as a company, would like to encourage
>bigints as
>> a sane reasonable default for primary keys for application models. If
>the
>> Django project agrees with this idea, that would mean that Django
>would
>> provide BigAutoField as the default for ‘id’ instead of AutoField.
>>
>>
>>
>> Rails made this change recently
>> , and has seen
>success
>> in doing so.
>>
>>
>>
>> I’m happy to provide the code to do this, but I wanted to discuss it
>here
>> before doing so, to hear what the general consensus was to the
>proposal of
>> such a change.
>>
>>
>>
>>
>>
>> Pros:
>>
>>-
>>
>>Users of Django, following the docs, won’t accidentally hit the
>int
>>overflow barrier.
>>-
>>
>>Encourages best-practices from the beginning.
>>-
>>
>>Bigints don’t take up much more storage than ints when using
>Postgres.
>>-
>>
>>In-line with other frameworks moving forward on this issue, like
>Rails
>>.
>>
>>
>>
>> Cons:
>>
>>-
>>
>>Backwards compatibility would need to be considered.
>>
>>
>> Why not UUID?
>>
>>
>>
>> I agree! I love using UUID for my primary keys, and I think a patch
>to
>> Django which provides an AutoUUIDField would be wonderful. However,
>there
>> are a few major drawbacks to making this the new default:
>>
>>
>>
>>1.
>>
>>It’s confusing to new users, would make onboarding process more
>>difficult.
>>2.
>>
>>UUID is difficult to implement in MySQL.
>>3.
>>
>>UUID has larger storage requirements.
>>4.
>>
>>Incrementing IDs are actually useful.
>>
>>
>>
>>
>> So, my proposal is to simply lift the int barrier to a bigint barrier
>for
>> new Django applications, and I think it will save a lot of developers
>a lot
>> of pain in the long run.
>>
>>
>>
>> Many thanks,
>>
>>
>>
>> Kenneth Reitz
>>
>> Heroku Python
>>
>> --
>> You received this message because you are subscribed to the Google
>Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it,
>send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to
>django-developers@googlegroups.com.
>> Visit this group at
>https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit https://groups.google.com/d/
>> msgid/django-developers/6fe3401c-4404-4bd8-9d22-
>> 58df95cd1348%40googlegroups.com
>>
>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Django developers  (Contributions to Django itself)" group.
>To unsubscribe from this group and stop receiving emails 

Re: Default to BigAutoField

[Kenneth Reitz]

However, it should also be noted that those same larger applications are the 
ones that are likely to run into this problem eventually, so perhaps forcing 
the migration is the best path moving forward. 

Interested in hearing thoughts about this.

--
Kenneth Reitz

> On Jun 9, 2017, at 3:54 PM, Kenneth Reitz  wrote:
> 
> My initial thought was to just have this apply to *new* Django applications, 
> if that's possible.
> 
> Running this migration could take quite some time on some larger apps, and 
> would require a non-trivial amount of time to execute. 
> 
> --
> Kenneth Reitz
> 
>> On Jun 9, 2017, at 3:53 PM, Tom Forbes > > wrote:
>> 
>> How would this work with generic foreign keys as well? Those migrations 
>> couldn't be automatic.
>> 
>> On 9 Jun 2017 20:43, "Jacob Kaplan-Moss" > > wrote:
>> I think this would be a good improvement, and I'd like to see it. I've been 
>> bitten by integers overflowing at least twice I can remember in my career, 
>> which is two times too many.
>> 
>> However, a major thing we'd have to work out is the upgrade path Consider a 
>> simple model:
>> 
>> class Person(Model):
>> name = CharField()
>> 
>> In Django 1.11, this actually generates a model with an integer `id` field. 
>> But in we change it, in Django vNext, that `id` field would "turn into" a 
>> bigint magically without the underlying table changes. That'd be confusing: 
>> you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't 
>> be. I think the migrations engine would detect this as a migration (?), so 
>> perhaps that's the path forward, but it could still be super-confusing. 
>> We've never shipped a release of Django that required a migration to _all_ 
>> your models.
>> 
>> Have you thought about what the upgrade path should look like, Kenneth?
>> 
>> Jacob
>> 
>> On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz > > wrote:
>> Dear Django Dev,
>>  
>> At Heroku, we have the privilege of seeing an extremely broad range of 
>> customers utilizing tools like Django to build their applications and 
>> companies. One of the things that we’ve seen customers hit, time and time 
>> again when using tools like Django, is integer overflows for primary keys. 
>> Their application starts behaving unpredictably once they reach the 
>> overflow, not even knowing such a constraint exists, and they often think 
>> the problem is with their database provider, rather than with their schema. 
>> Once they realize what is wrong, it’s a relatively trivial fix, but a 
>> migration can take several hours to complete, which results in unacceptable 
>> amounts of downtime.
>>  
>> Because of this, Heroku, as a company, would like to encourage bigints as a 
>> sane reasonable default for primary keys for application models. If the 
>> Django project agrees with this idea, that would mean that Django would 
>> provide BigAutoField as the default for ‘id’ instead of AutoField. 
>>  
>> Rails made this change recently 
>> , and has seen success in 
>> doing so. 
>>  
>> I’m happy to provide the code to do this, but I wanted to discuss it here 
>> before doing so, to hear what the general consensus was to the proposal of 
>> such a change. 
>>  
>>  
>> Pros:
>> Users of Django, following the docs, won’t accidentally hit the int overflow 
>> barrier. 
>> Encourages best-practices from the beginning. 
>> Bigints don’t take up much more storage than ints when using Postgres. 
>> In-line with other frameworks moving forward on this issue, like Rails 
>> .
>>  
>> Cons:
>> Backwards compatibility would need to be considered. 
>>  
>> Why not UUID?
>>  
>> I agree! I love using UUID for my primary keys, and I think a patch to 
>> Django which provides an AutoUUIDField would be wonderful. However, there 
>> are a few major drawbacks to making this the new default:
>>  
>> It’s confusing to new users, would make onboarding process more difficult. 
>> UUID is difficult to implement in MySQL. 
>> UUID has larger storage requirements. 
>> Incrementing IDs are actually useful. 
>>  
>> 
>> So, my proposal is to simply lift the int barrier to a bigint barrier for 
>> new Django applications, and I think it will save a lot of developers a lot 
>> of pain in the long run. 
>>  
>> Many thanks,
>>  
>> Kenneth Reitz
>> Heroku Python
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-developers+unsubscr...@googlegroups.com 
>> .
>> To post to this group, send email to django-developers@googlegroups.com 
>> 

Re: Default to BigAutoField

[Kenneth Reitz]

My initial thought was to just have this apply to *new* Django applications, if 
that's possible.

Running this migration could take quite some time on some larger apps, and 
would require a non-trivial amount of time to execute. 

--
Kenneth Reitz

> On Jun 9, 2017, at 3:53 PM, Tom Forbes  wrote:
> 
> How would this work with generic foreign keys as well? Those migrations 
> couldn't be automatic.
> 
> On 9 Jun 2017 20:43, "Jacob Kaplan-Moss"  > wrote:
> I think this would be a good improvement, and I'd like to see it. I've been 
> bitten by integers overflowing at least twice I can remember in my career, 
> which is two times too many.
> 
> However, a major thing we'd have to work out is the upgrade path Consider a 
> simple model:
> 
> class Person(Model):
> name = CharField()
> 
> In Django 1.11, this actually generates a model with an integer `id` field. 
> But in we change it, in Django vNext, that `id` field would "turn into" a 
> bigint magically without the underlying table changes. That'd be confusing: 
> you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't 
> be. I think the migrations engine would detect this as a migration (?), so 
> perhaps that's the path forward, but it could still be super-confusing. We've 
> never shipped a release of Django that required a migration to _all_ your 
> models.
> 
> Have you thought about what the upgrade path should look like, Kenneth?
> 
> Jacob
> 
> On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz  > wrote:
> Dear Django Dev,
>  
> At Heroku, we have the privilege of seeing an extremely broad range of 
> customers utilizing tools like Django to build their applications and 
> companies. One of the things that we’ve seen customers hit, time and time 
> again when using tools like Django, is integer overflows for primary keys. 
> Their application starts behaving unpredictably once they reach the overflow, 
> not even knowing such a constraint exists, and they often think the problem 
> is with their database provider, rather than with their schema. Once they 
> realize what is wrong, it’s a relatively trivial fix, but a migration can 
> take several hours to complete, which results in unacceptable amounts of 
> downtime.
>  
> Because of this, Heroku, as a company, would like to encourage bigints as a 
> sane reasonable default for primary keys for application models. If the 
> Django project agrees with this idea, that would mean that Django would 
> provide BigAutoField as the default for ‘id’ instead of AutoField. 
>  
> Rails made this change recently 
> , and has seen success in 
> doing so. 
>  
> I’m happy to provide the code to do this, but I wanted to discuss it here 
> before doing so, to hear what the general consensus was to the proposal of 
> such a change. 
>  
>  
> Pros:
> Users of Django, following the docs, won’t accidentally hit the int overflow 
> barrier. 
> Encourages best-practices from the beginning. 
> Bigints don’t take up much more storage than ints when using Postgres. 
> In-line with other frameworks moving forward on this issue, like Rails 
> .
>  
> Cons:
> Backwards compatibility would need to be considered. 
>  
> Why not UUID?
>  
> I agree! I love using UUID for my primary keys, and I think a patch to Django 
> which provides an AutoUUIDField would be wonderful. However, there are a few 
> major drawbacks to making this the new default:
>  
> It’s confusing to new users, would make onboarding process more difficult. 
> UUID is difficult to implement in MySQL. 
> UUID has larger storage requirements. 
> Incrementing IDs are actually useful. 
>  
> 
> So, my proposal is to simply lift the int barrier to a bigint barrier for new 
> Django applications, and I think it will save a lot of developers a lot of 
> pain in the long run. 
>  
> Many thanks,
>  
> Kenneth Reitz
> Heroku Python
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com
>  
> .
> For more 

Re: Default to BigAutoField

[Tom Forbes]

How would this work with generic foreign keys as well? Those migrations
couldn't be automatic.

On 9 Jun 2017 20:43, "Jacob Kaplan-Moss"  wrote:

> I think this would be a good improvement, and I'd like to see it. I've
> been bitten by integers overflowing at least twice I can remember in my
> career, which is two times too many.
>
> However, a major thing we'd have to work out is the upgrade path Consider
> a simple model:
>
> class Person(Model):
> name = CharField()
>
> In Django 1.11, this actually generates a model with an integer `id`
> field. But in we change it, in Django vNext, that `id` field would "turn
> into" a bigint magically without the underlying table changes. That'd be
> confusing: you'd expect the model to be "fixed" by pugrading to vNext, but
> it wouldn't be. I think the migrations engine would detect this as a
> migration (?), so perhaps that's the path forward, but it could still be
> super-confusing. We've never shipped a release of Django that required a
> migration to _all_ your models.
>
> Have you thought about what the upgrade path should look like, Kenneth?
>
> Jacob
>
> On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz  wrote:
>
>> Dear Django Dev,
>>
>>
>>
>> At Heroku, we have the privilege of seeing an extremely broad range of
>> customers utilizing tools like Django to build their applications and
>> companies. One of the things that we’ve seen customers hit, time and time
>> again when using tools like Django, is integer overflows for primary keys.
>> Their application starts behaving unpredictably once they reach the
>> overflow, not even knowing such a constraint exists, and they often think
>> the problem is with their database provider, rather than with their schema.
>> Once they realize what is wrong, it’s a relatively trivial fix, but a
>> migration can take several hours to complete, which results in unacceptable
>> amounts of downtime.
>>
>>
>>
>> Because of this, Heroku, as a company, would like to encourage bigints as
>> a sane reasonable default for primary keys for application models. If the
>> Django project agrees with this idea, that would mean that Django would
>> provide BigAutoField as the default for ‘id’ instead of AutoField.
>>
>>
>>
>> Rails made this change recently
>> , and has seen
>> success in doing so.
>>
>>
>>
>> I’m happy to provide the code to do this, but I wanted to discuss it here
>> before doing so, to hear what the general consensus was to the proposal of
>> such a change.
>>
>>
>>
>>
>>
>> Pros:
>>
>>-
>>
>>Users of Django, following the docs, won’t accidentally hit the int
>>overflow barrier.
>>-
>>
>>Encourages best-practices from the beginning.
>>-
>>
>>Bigints don’t take up much more storage than ints when using
>>Postgres.
>>-
>>
>>In-line with other frameworks moving forward on this issue, like Rails
>>.
>>
>>
>>
>> Cons:
>>
>>-
>>
>>Backwards compatibility would need to be considered.
>>
>>
>> Why not UUID?
>>
>>
>>
>> I agree! I love using UUID for my primary keys, and I think a patch to
>> Django which provides an AutoUUIDField would be wonderful. However, there
>> are a few major drawbacks to making this the new default:
>>
>>
>>
>>1.
>>
>>It’s confusing to new users, would make onboarding process more
>>difficult.
>>2.
>>
>>UUID is difficult to implement in MySQL.
>>3.
>>
>>UUID has larger storage requirements.
>>4.
>>
>>Incrementing IDs are actually useful.
>>
>>
>>
>>
>> So, my proposal is to simply lift the int barrier to a bigint barrier for
>> new Django applications, and I think it will save a lot of developers a lot
>> of pain in the long run.
>>
>>
>>
>> Many thanks,
>>
>>
>>
>> Kenneth Reitz
>>
>> Heroku Python
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%
>> 40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to 

Re: Default to BigAutoField

[Jacob Kaplan-Moss]

I think this would be a good improvement, and I'd like to see it. I've been
bitten by integers overflowing at least twice I can remember in my career,
which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a
simple model:

class Person(Model):
name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field.
But in we change it, in Django vNext, that `id` field would "turn into" a
bigint magically without the underlying table changes. That'd be confusing:
you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't
be. I think the migrations engine would detect this as a migration (?), so
perhaps that's the path forward, but it could still be super-confusing.
We've never shipped a release of Django that required a migration to _all_
your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz  wrote:

> Dear Django Dev,
>
>
>
> At Heroku, we have the privilege of seeing an extremely broad range of
> customers utilizing tools like Django to build their applications and
> companies. One of the things that we’ve seen customers hit, time and time
> again when using tools like Django, is integer overflows for primary keys.
> Their application starts behaving unpredictably once they reach the
> overflow, not even knowing such a constraint exists, and they often think
> the problem is with their database provider, rather than with their schema.
> Once they realize what is wrong, it’s a relatively trivial fix, but a
> migration can take several hours to complete, which results in unacceptable
> amounts of downtime.
>
>
>
> Because of this, Heroku, as a company, would like to encourage bigints as
> a sane reasonable default for primary keys for application models. If the
> Django project agrees with this idea, that would mean that Django would
> provide BigAutoField as the default for ‘id’ instead of AutoField.
>
>
>
> Rails made this change recently
> , and has seen success
> in doing so.
>
>
>
> I’m happy to provide the code to do this, but I wanted to discuss it here
> before doing so, to hear what the general consensus was to the proposal of
> such a change.
>
>
>
>
>
> Pros:
>
>-
>
>Users of Django, following the docs, won’t accidentally hit the int
>overflow barrier.
>-
>
>Encourages best-practices from the beginning.
>-
>
>Bigints don’t take up much more storage than ints when using Postgres.
>-
>
>In-line with other frameworks moving forward on this issue, like Rails
>.
>
>
>
> Cons:
>
>-
>
>Backwards compatibility would need to be considered.
>
>
> Why not UUID?
>
>
>
> I agree! I love using UUID for my primary keys, and I think a patch to
> Django which provides an AutoUUIDField would be wonderful. However, there
> are a few major drawbacks to making this the new default:
>
>
>
>1.
>
>It’s confusing to new users, would make onboarding process more
>difficult.
>2.
>
>UUID is difficult to implement in MySQL.
>3.
>
>UUID has larger storage requirements.
>4.
>
>Incrementing IDs are actually useful.
>
>
>
>
> So, my proposal is to simply lift the int barrier to a bigint barrier for
> new Django applications, and I think it will save a lot of developers a lot
> of pain in the long run.
>
>
>
> Many thanks,
>
>
>
> Kenneth Reitz
>
> Heroku Python
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/6fe3401c-4404-4bd8-9d22-
> 58df95cd1348%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAK8PqJELsQdpKEcg-a_iEKKwoYkoGmc8pikFM-hgG4g0CZE8XQ%40mail.gmail.com.
For more options, visit