When I wrote small I was thinking along the lines of clearing up particular methods to ensure they're not breaking semi-formal contracts. Layering on a new abstraction to fields is quite a bit bigger, but is still small relative to the goal of "decoupling the ORM". If you can show a clear benefit for that direction and it doesn't break backwards compatibility then I think the idea would be welcomed.
On Sunday, 20 December 2015 19:46:44 UTC+11, Samuel Bishop wrote: > > I was more referring to 'above' in a broad way, as the fields are > constructed from the data returned by the ORM when building the model. > which should also be above the ORM. > > Pulling open the Django code in PyCharm and taking a look at things, with > an eye towards "lots of smaller incremental improvements" has me > wondering... how small is small? > > As part of restructuring the layers, it would make sense to create a > higher level module, stub the 'new' models and fields into the old > locations for backwards compatibility, and separate the ORM related and non > ORM related parts of the model and field classes, placing the non ORM part > in the new module, and likely keeping the rest in the current place since > `django.db.models` since 'django database models' feels like the right > place for database specific parts of model code. Splitting it like that > would allow the placement of our new 'chose your back end' logic to exist > between the non ORM parts and the ORM parts by the use of dynamic > subclassing based on configuration options much as we do inside the ORM. > > But while that could happen with little new code needing to be written, > since its primarily about restructuring the existing code. It doesn't > however feel like a small change, and no matter how I cut it up, at some > point there would likely need to be a fairly big "cut over" PR. Not really > in the spirit of lots of small increments. > > On Sunday, 20 December 2015 07:31:48 UTC+8, Josh Smeaton wrote: >> >> As far as I'm concerned, anything we can do to simplify and decouple >> abstraction layers is a good thing and would be welcomed, with the usual >> caveats of backwards compatibility in public and pseudo-public APIs. >> >> models.query is a thin layer over models.sql.query. That's a good thing. >> Ideally, models.query should be able to use any sql.query underneath, but >> there's currently no good way to swap out implementations. I think >> simplifying models.query so that it is *only* a thin wrapper over the >> backend query class would be a great first start. >> >> I think there's a tonne of room for improvement with sql.query also. It >> does way more than just store the current state of the query. I feel a lot >> of the functionality of sql.query should be pushed down to the Compiler >> class. >> >> As far as fields themselves go, they're distinct from the >> queryset/query/compiler layer, so I wouldn't say they are "above" or >> "below" queryset itself. Perhaps fields would benefit from a similar >> abstraction where there is a very thin layer up top, and a "backend" >> implementation that does the heavy lifting. Again though, figuring out a >> nice way to swap out implementations would be a challenge, and preserving >> backwards compatibility of fields is going to be even harder than that of >> query and compiler, because fields are most definitely public/pseudo-public. >> >> But going back to my original point, we'd welcome lots of smaller >> incremental improvements to the abstraction layers. Fields are going to be >> hard to improve though. They offer so many different methods that aren't >> well documented but are definitely relied on from all backends. >> >> Cheers >> >> >> On Sunday, 20 December 2015 02:39:13 UTC+11, Samuel Bishop wrote: >>> >>> You've raise a very good point, one that has been on my mind the entire >>> time I've been exploring this. >>> Much of Django has been designed 'on the back of' the ORM, reliant on >>> its limitations and without need to go beyond the scope of what it can >>> provide, its very probable that over time this has introduced some failures >>> to fully adhere to the best possible separation of concerns. >>> >>> Autofield does seem to suffer from this a little with its assumption of >>> Integer only keys. >>> Fundamentally, when it comes to letting the database do the work for us, >>> Django doesn't need a lot of code to make a field... I've seen alternate >>> (non primary key) autofields as short as 3 lines. If we took the simplest >>> approach possible, a 'project wide setting'. Such a setting could 'toggle' >>> between integer and string/UUID behavior everywhere we need to introduce >>> changes. However I feel that may be too simplistic an approach if I want to >>> try and get broader support for NoSQL/Alternate DB support via the approach >>> I'm so determined to try and build. >>> >>> My biggest worry is really, just how many places check the DB >>> parameters. A cursory reading of the base Field class, has literally dozens >>> of references to both the default connection wrapper object `connection`, >>> and to the database connectionhandler object `connections`. 55 references >>> at last check in `django.db.models.fields` alone. It worries me that I may >>> still need to create essentially a fake database backend full of dozens of >>> stub entries to satisfy the introspection being performed by various >>> classes despite using a queryset that never even makes a database call. If >>> anyone with more knowledge about this wants to comment, it would be most >>> useful. Currently, I'm operating under the assumption, educated at least in >>> part by studying the DB connection classes, testing and debugging various >>> alternative database backends, and debugging my way through ORM calls,that >>> the queryset interface is the uppermost level of ORM interfacing code. In >>> principle nothing above the queryset should rely on any particular >>> database, or any database at all. If there are parts of Django "above" the >>> queryset that rely on knowing things about the database, how defective >>> should we consider this? Should I be temporarily working around such >>> issues, commenting things out, logging bugs etc, or implementing >>> workarounds in my own queryset... such as the aforementioned 'fake database >>> backend'. >>> >>> >>> On Saturday, 19 December 2015 01:42:40 UTC+8, Marc Tamlyn wrote: >>>> >>>> I agree that the current uuidfield is a simplistic solution to be used >>>> as autofield, and having a way to specify your autofield type globally >>>> would be good, equally something like a biginteger field. The complexity >>>> involved with doing this (custom RETURNING statements, database level >>>> defaults, integration with migrations) is an order of magnitude more >>>> complex than the solution committed. I knew that was the intention of the >>>> original issue but it was also why it hadn't been solved for the "simple" >>>> case. >>>> >>>> I did spend some time looking at the more complete solution and >>>> honestly could not work out how to approach it. I found if hit many more >>>> parts of the internals of the ORM than I expected. >>>> On 18 Dec 2015 12:32 a.m., "Curtis Maloney" <cur...@tinbrain.net> >>>> wrote: >>>> >>>>> I've identified one new "issue". >>>>>> There is an implicit assumption that primary keys are useful for >>>>>> ordering by the current QuerySet API methods `.first()` and `.last()`. >>>>>> >>>>> >>>>> I believe the case here is that first and last are meaningless without >>>>> an ordering, so in lieu of any programmer supplied ordering, a "fallback" >>>>> consistent ordering of 'pk' is used. >>>>> >>>>> Remember the DBMS is under no obligation to return a query in >>>>> consistent order without an ORDER BY clause. >>>>> >>>>> So, without an ordering, two successive calls to first() [or last()] >>>>> may, in fact, return different records even without modifications to the >>>>> table. >>>>> >>>>> It's not expected people should _rely_ on the ordering of PK, and, >>>>> indeed, it's frequently recommended against inferring any meaning from PK >>>>> values [sqlite, IIRC, assigns them pseudo-ramdomly] >>>>> >>>>> That said, the assumption that a PK is sortable and will provide a >>>>> deterministic ordering shouldn't be much to ask, surely? >>>>> >>>>> -- >>>>> Curtis >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Django developers (Contributions to Django itself)" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to django-develop...@googlegroups.com. >>>>> To post to this group, send email to django-d...@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/django-developers. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/django-developers/567353FA.6030105%40tinbrain.net >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0c141f72-68ff-4bc8-a2f6-b0122445a7bc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
