Re: Proposal: use SQLAlchemy Core for query generation

[Asif Saifuddin]

Hi Luke,

I have been thinking and researching a lot recently about SQLalchemy and 
django integration.

Now SQLalchemy provides a devlarative orm extension too quite identical to 
django orm[not same], django has also refactored a lot in db internals with 
model meta refactore, expressions, new migrations and many stuff.

There would be 2 new approach for your proposal--

1) Probably the people used to/willing to use sqlalchemy's declarative orm 
extension will use that and there will be needed to create new model meta 
API driven abstraction layer which will help to show sqlalchemy generated 
models in django admin and django form/model form.
  * migrations would be handled by alaembic/sqlalchemy-migrate/whatever 
best available

2) Django Model ORM will be  rewritten in a 3rd party project as a parallel 
counterpart/equivalent to SQLalchemy's declarative ORM extension which will 
work like django model//migrations/command based normal djangoish workflow 
but under the hood will be used by sqlalchemy core/3rd party tools.

I'm willing to address the both issue in a solid 3rd party package. In both 
case propably django internal too needed to be ironed for edge cases.

I'm trying to tracking this thoughts in a git repo and also looked for some 
older attempts to understand the implementation way.

Looking forward to hear your thoughts.


Regards,

Asif



On Saturday, June 30, 2012 at 8:22:21 PM UTC+6, Luke Plant wrote:
>
> Hi all, 
>
> A good while back I put forward the idea of using SQLAlchemy Core in 
> Django [1]. Having had more experience working with SQLAlchemy, I'm 
> putting that idea forward as a formal proposal, as I mentioned in a more 
> recent thread here. 
>
> Apologies in advance for the length! I've included a few 'TL;DR' 
> summaries and headings in the different sections which you might want to 
> scan first. 
>
> === Proposal === 
>
> We should re-write our query generation code to use SQLAlchemy Core. 
> This includes DDL queries as well as DML (e.g. CREATE TABLE as well as 
> SELECT, INSERT etc). 
>
> This would also involve replacing our database connection objects with 
> SQLAlchemy's. In this proposal, our high-level ORM, with model 
> definition and query API, would remain the same - we wouldn't be using 
> the SQLAlchemy ORM. 
>
> This is a "Django 2.0" proposal i.e. not immediate future, and not fully 
> backwards compatible. 
>
> === Background - Django 2.0 philosophy === 
>
> TL;DR: "evolution not revolution, some backwards incompatibilities" 
>
> Although we haven't really discussed the timing of Django 2.0, or its 
> philosophy, I think we should be doing. My own assumption is that it 
> should be evolution, not revolution, similar to Python 3, where we break 
> stuff that has dogged us in the past, and will hamper us going forward, 
> but don't make radical changes where not necessary. This proposal fits 
> that basic assumption. 
>
> Also, in Django to date we've eschewed external dependencies. That has 
> been partly due to the poor and confusing state of Python packaging, 
> which is hopefully improving. I think it will make us a very poor Python 
> citizen if we don't reverse this policy at some point, and Django 2.0 is 
> an obvious point to do it. 
>
> This proposal does not represent a move away from being a 'full stack' 
> framework. "Full stack" does not mean "no dependencies". 
>
> Our current recommended installation method is via pip [2], and we would 
> be doing our users a disservice to recommend otherwise. Installation via 
> pip actually makes the instructions shorter - manual methods require 
> things like removing old versions manually - and for anything beyond 
> trivial use of Django you have to know pip anyway. 
>
> So, with our recommended installation method, adding a dependency 
> doesn't make things any more difficult at all. 
>
> === Background - ORM philosophy === 
>
> TL;DR: "Let's make Django's DB layer the best it can be for relational 
> databases." 
>
> Whether we call it "the ORM" or "the model layer" as some people prefer, 
> I think it's fairly certain that the overwhelming majority of our users 
> are using relational databases. 
>
> Many of the things that make Django a compelling choice, 
> including the admin and re-usable apps, either don't work or are of 
> little use if you are not using a relational database. 
>
> So my philosophy is that we should aim to provide a really excellent 
> ORM that will take users as far as possible. 
>
> This doesn't preclude having non-relational support in Django. But 
> it seems very strange to make that the focus, when we've had 
> little-to-no support for it so far, or to allow that support to limit 
> how well we can cater for the 99%. 
>
> === Motivation === 
>
> == Motivation 1: Django's ORM leaves you high and dry when you reach its 
> limits. 
>
> While the ORM can do a surprising number of queries, there are plenty it 
> can't, and in all the medium-to-large projects I've worked on I've go

Re: ORM difficulties

[Josh Smeaton]

>
> Also, some random feature requests:
>
>- .values(‘stuff’, my_thing=Coalesce(‘thing’, ‘stuff’)) should work
>
>  I thought this was already tracked but I couldn't find an existing 
ticket. So I created it: https://code.djangoproject.com/ticket/25871
Feel free to leave comments there if you feel that's a good idea, and to 
discuss some API ideas (like how to preserve existing **kwargs in values()).

>
>- there should be a provided Year and Month functions to extract 
>years/months from date fields
>
> It's coming in 1.10 https://github.com/django/django/pull/5683, although 
you can already sort of use the private versions (which will be moved in 
1.10 so be 
careful) 
https://github.com/django/django/blob/6f229048ddd8c7347ff60dddfb9121e6021c7b2e/django/db/models/lookups.py#L511

As far as grouping semantics go, I'm open to discussion! If someone was to 
put forward a proposal that could work and wouldn't unnecessarily burden 
users that aren't intimately familiar with SQL, then that would likely be 
welcome. Anssi mentioned exposing the underlying grouping method/dict which 
may be a good idea, but we'd like to avoid messing around with internal 
state too much.

Personally, I don't find grouping to be too much of a problem. Django 
generates the correct grouping based on what you've limited the columns to 
in your `values()` clause. There are outliers which we can maybe defend 
against with docs. The most unintuitive thing is adding `Meta: ordering` 
columns to the group by list, but I think that changed recently, or was 
just added?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d95be9df-2285-4105-bd22-99ad7cc016e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: annoyance with Python 3.2 support in Django 1.8

[Tim Graham]

No we haven't done that before. I think advertising it in the blog post and 
release notes would be enough. In particular, I'd like to advertise it as a 
tentative plan just in case someone comes along after we advertise the plan 
more widely and offers a compelling reason to continue Python 3.2 support.

On Thursday, December 3, 2015 at 8:08:12 PM UTC-5, Dan Stephenson wrote:
>
> Do we currently raise any warnings/exceptions in cases where Python 
> support has / or is about to be dropped (particularly mid LTS)..   As a 
> suggestion, I was thinking it could be helpful to people affected we raised 
> exception msg indicating the last Django version to support their current 
> Python version?I'd be happy to build if thought useful.
>
>
>
>
> On Thursday, 3 December 2015 23:30:42 UTC, Chris Streeter wrote:
>>
>> Donald could probably provide more information, but this post from April 
>> shows the Python 3.2 numbers downloading from PyPI are constant, and pretty 
>> small [https://caremad.io/2015/04/a-year-of-pypi-downloads/] His take 
>> was that CI systems (like Django's!) were doing most of the Python 3.2 
>> package downloading.
>>
>> On Thu, Dec 3, 2015 at 2:18 PM, Josh Smeaton  wrote:
>>
>>> I agree with Tim. Unless someone puts their hand up to say they 
>>> definitely require python 3.2 support for 1.8, I think it makes sense to 
>>> drop support in the next dot release of 1.8. 3.2 isn't an easy python to 
>>> find in the wild as far as I know, so I'd be surprised if there was any 
>>> real support for it on 1.8 by users.
>>>
>>> On Friday, 4 December 2015 02:50:24 UTC+11, Tim Graham wrote:

 No, using pypy3 doesn't make things easier. There are a handful of test 
 failures with pypy3 and it doesn't solve the issue that 
 unittest-xml-reporting doesn't work with Python 3.2.

 Issues aside, the main thing I'm trying to find out is, are we 
 providing any substantial value supporting Django on an unsupported 
 version 
 of Python? So far no one has indicated "yes". If you care about Django 
 security updates, shouldn't you care about Python security updates too?

 On Wednesday, December 2, 2015 at 5:22:46 PM UTC-5, Shai Berger wrote:
>
> On Wednesday 02 December 2015 21:05:00 Tim Graham wrote: 
> > 
> > Given that no one reading this indicated that they plan a long-term 
> > deployment of Python 3.2, how about if in the next 1.8.x release we 
> > advertise that Python 3.2 support for Django 1.8 will end January 1, 
> 2017? 
> > (we won't break anything intentionally after that, but we won't have 
> to 
> > worry about testing and can spin down our 12.04 machine before it's 
> EOL a 
> > few months later) 
> > 
>
> Since you brought the issue up yourself -- shouldn't we "swap" PyPy3 
> for 
> Python 3.2? Would that make running tests on ubuntu 14.04 easier? 
>
> Just a half-baked thought, 
>
> Shai. 
>
 -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/0a0631c3-4733-46fc-8884-c4050c8b8701%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5d501c8f-a413-4854-bfae-d3d63159c144%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Proposal to merge django-csp into contrib

[Thomas Grainger]

Now that the django admin supports the Content-Security-Policy header I 
think merging django-csp into contrib would be a good fix for 
https://code.djangoproject.com/ticket/15727 

It might also be an idea to wait for Mozilla to also respond to the 
proposal of this merger.

See my PR here: https://github.com/django/django/pull/5776

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/807ded1c-8d1e-45e1-a705-ac316ca63a20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fellow Report - December 5, 2015

[Tim Graham]

Triaged

---

https://code.djangoproject.com/ticket/25808 - Setting to allow 
django.test.Client default requests to follow=True (wontfix)

https://code.djangoproject.com/ticket/25840 - cache.get_or_set() works 
incorrectly with DummyCache backend (accepted)

https://code.djangoproject.com/ticket/25844 - django-1.9 fails test under 
pypy (invalid)

https://code.djangoproject.com/ticket/25831 - 
django.views.generic.edit.FormMixinBase : get_form_with_form_class wrapping 
preventing custom method profiles for get_form() (wontfix)

https://code.djangoproject.com/ticket/25845 - False timezone offset 
warnings in admin if a custom base template doesn't set 
data-admin-utc-offset (accepted)

https://code.djangoproject.com/ticket/25849 - pypy3 ImportError field_spec 
(wontfix)

https://code.djangoproject.com/ticket/25870 - Wrong position of Action 
button in admin when using filters (accepted)

Authored



https://github.com/django/djangoproject.com/pull/565 - Added "Download 
Donor Report" admin action.

https://github.com/django/djangoproject.com/pull/559 - Moved fundraising 
page to /fundraising/

https://github.com/django/djangoproject.com/pull/560 - Removed ForeignKeys 
to Campaign.

https://github.com/django/djangoproject.com/pull/561 - Removed the Campaign 
model.

https://github.com/django/djangoproject.com/pull/568 - Fixed #566 -- Made 
"Cancel this donation" button use POST.

https://github.com/django/djangoproject.com/pull/570 - Fixed #567 -- Added 
DjangoHero.location

https://github.com/django/django/pull/5772 - Refs #25677 -- Skipped an i18n 
test on older gettext versions.

https://github.com/django/django/pull/5777 - Fixed #25860 -- Documented a 
transaction leak possiblity in TestCase.

Reviewed/committed

--

https://github.com/django/django/pull/5743 - Fixed #25827 -- Removed extra 
spacing in admin's DateTimeField.

https://github.com/django/django/pull/5541 - Fixed #25667 -- Fixed 
admindocs initial_header_level.

https://github.com/django/django/pull/5755 - Fixed many spelling mistakes 
in code, comments, and docs.

https://github.com/django/django/pull/5760 - Refs #25659 -- Added missing 
docs for Difference/Intersection/SymDifference functions.

https://github.com/django/django/pull/5736 - Fixed #25825 -- Implemented 
__ne__ for template Origin

https://github.com/django/django/pull/5759 - Fixed #25851 -- Updated link 
to deprecated assignment tag docs.

https://github.com/django/django/pull/5721 - Fixed #25761 -- Added 
__cause__.__traceback__ to reraised exceptions.

https://github.com/django/django/pull/5708 - Fixed #25797 -- Fixed regex 
for getting units from SRS WKT.

https://github.com/django/django/pull/5739 - Fixed #25820 -- Allowed 
whitespace in admin's calendar.js month/weekday names.

https://github.com/django/django/pull/5645 - Fixed #13774 -- Added 
models.Field.rel_db_type().

https://github.com/django/django/pull/5756 - Fixed #25840 -- Fixed 
BaseCache.get_or_set() on the DummyCache backend.

https://github.com/django/django/pull/5768 - Fixed #25740 -- Documented 
GEOSGeometry operators.

https://github.com/django/django/pull/5567 - Fixed #25165 -- Removed inline 
JavaScript from contrib.admin.

https://github.com/django/django/pull/5773 - Fixed #25547 -- Made 
Model.refresh_from_db() update FileField's instance.

Reviews of core dev work


https://github.com/django/django/pull/5771 - Fixed #25867 -- Fixed a system 
check crash with nested ArrayFields.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/da758cee-6a1f-4d3c-9893-d5b77580716a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM difficulties

[Aymeric Augustin]

Hello,

> On 5 déc. 2015, at 10:28, Raphael Gaschignard  wrote:
> 
>   A behaviour that exemplifies this is that 
> queryset.annotate(foo=thing).annotate(bar=other_thing) is not the same as 
> queryset.annotate(foo=thing, bar=other_thing) given certain things. This goes 
> against the intuitive interpretation of the queryset API IMO.

Isn't this the same difference as between 
queryset.filter(foo__bar=bar).filter(foo__baz=baz) and 
queryset.filter(foo__bar=bar, foo__baz=baz)? I’m asking because if the answer 
is “yes”, then we want to keep the two APIs consistent and we can’t quite 
change how chained filter calls set up joins.

I don’t quite have a good answer about your other questions… I know that some 
people will say “stick to raw SQL”. That’s even what the documentation advises. 
However it appears that the ORM’s ability to compose querysets is so powerful 
that it outweighs the greater expressivity of SQL in many practical cases. When 
writing complex aggregations, I’ve had more luck trying to express the logic 
with Django’s APIs than trying to produce a particular SQL query. It was still 
very frustrating and mostly based on trial and error.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9420983C-F884-4720-952C-76C70B9FB469%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

ORM difficulties

[Raphael Gaschignard]

Hi list, 

  I want to preface this by saying I’m really glad to see the ORM where it 
is in 1.8, it’s gotten really far and I now think it’s not hopeless to 
imagine writing complex things in the ORM…


   So earlier this week I wrote a bit of a rant in the bug tracker about 
how annotate is confusing (https://code.djangoproject.com/ticket/25834).


  A short summary is that annotate and values (with a spattering of order_by) 
are basically what decides grouping, but there’s a lot of trial and error 
involved in getting the right grouping with a tricky interaction between 
the two.


  A behaviour that exemplifies this is that 
queryset.annotate(foo=thing).annotate(bar=other_thing) is *not* the same as 
queryset.annotate(foo=thing, 
bar=other_thing) given certain things. This goes against the intuitive 
interpretation of the queryset API IMO.


  I think there should be some update to the API to render grouping more 
explicit. Absent that, given that the docs are 

now oriented in a sort of “You shouldn’t need extras anymore” fashion, 
there really should be a “SQL to ORM” migration guide/cookbook to point out 
explicitly how to go from SELECT stuff FROM table GROUP BY properties to 
the right annotate/values. I found what jarshwah pointed out in the ticket 
was really helpful.


  Also, some random feature requests:

   - .values(‘stuff’, my_thing=Coalesce(‘thing’, ‘stuff’)) should work
   - there should be a provided Year and Month functions to extract 
   years/months from date fields

  Anyways I wanted to share this experience so that anyone who has the 
courage to right new docs/continue evolving things can know of at least one 
team’s difficulties.


 Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8d175352-d15e-4dc2-a471-014d6b0bc3e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Referrer Policy Delivery, Django shouldn't do strict referrer check anymore?

[Collin Anderson]

Basically, the origin check would only be useful for safari (in this case).
Everywhere else we'd still need to rely on the referrer header.

On Sat, Dec 5, 2015 at 3:42 AM, Florian Apolloner 
wrote:

>
>
> On Friday, December 4, 2015 at 8:03:45 PM UTC+1, Flávio Junior wrote:
>>
>> I can create a ticket suggesting Django to check Origin header before
>> checking Referer. Or do you want to create that Collin?
>>
>
> I think Firxfox does not send the origin header ever yet, do you have any
> docs on that (Aside from CORS with Ajax from the looks of it)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1cb9d411-8247-4e93-8120-5d043869c001%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFO84S6%3Dww8-Veiy66nhAM9PUY-Z4w02imB50Ho89ytZexWciA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Referrer Policy Delivery, Django shouldn't do strict referrer check anymore?

[Florian Apolloner]

On Friday, December 4, 2015 at 8:03:45 PM UTC+1, Flávio Junior wrote:
>
> I can create a ticket suggesting Django to check Origin header before 
> checking Referer. Or do you want to create that Collin?
>

I think Firxfox does not send the origin header ever yet, do you have any 
docs on that (Aside from CORS with Ajax from the looks of it)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1cb9d411-8247-4e93-8120-5d043869c001%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.