Re: Django ORM support for NoSql databases

[Russell Keith-Magee]

On Thu, Dec 19, 2013 at 1:47 AM, Javier Guerra Giraldez
wrote:

> On Wed, Dec 18, 2013 at 12:18 PM,  
> wrote:
> >
> > Wouldn't an easy (i.e. straightforward) solution be to add an Django
> "ODM"
> > that mirrors the ORM wherever it makes sense?  This sounds pretty close
> to
> > your second solution, except choosing SQL vs NoSQL means users make a
> more
> > explicit choice whether to use the ODM API vs the ORM API.
>
> I think it would be very straightforward to do that at the public API
> level, but the ModelForm and the admin use lots of undocumented _meta
> fields which would have to be mimicked too.
>
> Still, i'm convinced that this would be the best answer, and a formal
> definition of _meta is a big plus, maybe even proportional to the
> effort needed.
>

Agreed. Independent of whether NoSQL gets added to core, _meta would
definitely benefit from an implementation cleanup and formalisation as a
backwards-compatible API.

(Actually… this might make a good GSoC project…)

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJxq84-%2BWSpoWT9-ogn_WxCnZdhafKKsirKaoNqu64A60dpxTw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Django ORM support for NoSql databases

[Russell Keith-Magee]

On Thu, Dec 19, 2013 at 1:18 AM, wrote:

>
>
> On Tuesday, December 17, 2013 8:12:43 PM UTC-6, Russell Keith-Magee wrote:
>>
>>
>> My claim is that complete abstraction of the data store shouldn't be the
>> goal. What we should be aiming for is sufficient API compatibility to allow
>> for two things:
>>
>>  * ModelForms wrapping a model from a NoSQL data store
>>  * An admin representation of a NoSQL data model.
>>
>> These two functions don't require complex relations. They require
>> relatively simple CRUD operations on a single object; they require an
>> implementation of filter, but not an implementation that allows joins of
>> any kind inside a filter clause. Yes, this will cut off some functionality
>> -- some custom filters, for example, won't be possible. It might be
>> necessary to remove total object counts in admin ListViews -- but then,
>> that would be a good performance boost for PostgreSQL as well.
>>
>> But broadly speaking, I see no reason why it shouldn't be possible to put
>> store a data model in NoSQL, and visualise the contents of that model in
>> admin. And you should be able to throw up a Form wrapping that data model.
>>
>> I see two ways to achieve this goal. The first is to put an alternate
>> backend into Django's existing ORM. This is the approach attempted by Alex
>> in his GSoC project. It's possible, and actually doesn't require that much
>> work; but there are a couple of unresolved issues, mostly around handling
>> of non-integer automatic primary keys. These are problems that we might be
>> worth addressing anyway, because UUID-based auto primary keys would be a
>> worthwhile extension to SQL data stores.
>>
>> The second approach is to produce a duck type compatible model API. There
>> aren't (or shouldn't be) any instance checks tied to
>> django.db.models.Model; as long as your NoSQL layer implements the same
>> API, you should be able to drop it into the admin or a ModelForm. Taking
>> this approach would have the side effect that we'd have to clean up the
>> formal definition of _meta, which in itself would be a nice goal.  Added
>> benefit -- this approach doesn't actually require any changes to core
>> itself -- the ducks can be completely external to Django. The only changes
>> to core would be whatever cleanups and documentation the project identifies.
>>
>> The other, unrelated problem isn't technical at all -- its that that
>> enthusiasm for this problem has pretty much dried up in the core team
>> AFAICT. I certainly don't have any call for using a NoSQL store in my life
>> at the moment. However, if a ready-to-use implementation of either of these
>> approaches were to land on our doorstep, I wouldn't resist them being added
>> to core.
>>
>
> Wouldn't an easy (i.e. straightforward) solution be to add an Django "ODM"
> that mirrors the ORM wherever it makes sense?  This sounds pretty close to
> your second solution, except choosing SQL vs NoSQL means users make a more
> explicit choice whether to use the ODM API vs the ORM API.
>

Essentially, what you've described here *is* the second solution, with the
caveat that in order to maintain basic compatibility with the internals of
Django, you intentionally maintain API compatibility between the ORM and
ODM with some key aspects of the ORM API - so, for example,
MyMongoDBModel.objects.filter() would work, within certain constraints (for
example, as long as you only reference local fields).

A richer ODM/document API that better exposes the properties of
document-based stores would expand on these basics, but the basics are
needed so that it's drop-in compatible with the basics of ModelForm and
admin.

However, this doesn't need to be added to Django's Core, and I'd resist
adding it to core unless it can actually demonstrate that it's a genuine
abstraction (in the same way that the ORM abstracts the differences between
SQLite, MySQL and PostgreSQL). I'm not currently confident that NoSQL
stores are at a sufficient level of maturity that this sort of abstraction
is possible (or plausible) for anything beyond trivial examples. Remember,
SQL has been a work in progress for 20 years as a *published* standard (and
another 20 prior to that as an informal one) - the common ground in SQL is
well understood at this point. The same isn't true of NoSQL.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJxq848B38bGLBy-F9mg98WgwpT0e-sHbnMEyrPE_PMN72RpXQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: LiveServerTestCase, and override_settings(DEBUG=True)

[Russell Keith-Magee]

On Wed, Dec 18, 2013 at 11:57 PM, Harry Percival
wrote:

> Django's test runner overrides your settings to force DEBUG to be True,
> which I understand the intention behind, but it is occasionally annoying.
> One solution for those cases is to use `override_settings`, but that has
> very weird effects when using `LiveServerTestCase`.
>

I presume this is just a typo, but for clarity - it's the other way around.
TestRunner forces DEBUG=False, with the theory being that you should be
testing "production" behaviour as much as possible.


> Minimal repro:
>
> django-admin.py startproject myproj
> cd myproj
> vi tests.py
>
> In tests.py:
>
> from django.test import LiveServerTestCase
> from selenium import webdriver
> from django.test.utils import override_settings
>
> @override_settings(DEBUG=True)
> class MinimalTest(LiveServerTestCase):
>
> def setUp(self):
> self.browser = webdriver.Firefox()
> self.browser.implicitly_wait(3)
>
> def tearDown(self):
> self.browser.quit()
>
>
> def test_admin_site_is_there(self):
> self.browser.get(self.live_server_url + '/admin')
> self.assertIn('Django administration',
> self.browser.find_element_by_tag_name('body').text)
>
> Change DEBUG=True to DEBUG=False and this passes.  But, as-is, you get:
>
>   File "/tmp/myproj/test1.py", line 18, in test_admin_site_is_there
> self.assertIn('Django administration',
> self.browser.find_element_by_tag_name('body').text)
> AssertionError: 'Django administration' not found in 'Page not found
> (404)\nRequest Method: GET\nRequest URL: http://localhost:8081/admin\n"admin;
> does not exist\nYou\'re seeing this error because you have DEBUG = True in
> your Django settings file. Change that to False, and Django will display a
> standard 404 page.'
>
> I'm just using the admin URL because it's one that is activated by
> default.  Normal URLs, eg a home page view, also return 404s.  What's going
> on?
>

Ok - the reason for this is a little complicated.

The LiveTestServer is actually three media servers, layered over each other
(like an onion. Or a parfait :-). The outer layer is the static files
server, the middle layer is the media file server, and the inner layer is
the actual WSGI server providing your URLs.

Starting from the outside and moving in, each layer does a check to see
"should I serve this URL"; if it does, it returns the appropriate page/file.

However, if the file *doesn't* match, the behaviour of these layers is
different depending on the value of DEBUG. If DEBUG=False, it *raises* a
Http404; if DEBUG=True, it *returns* a technical 404 response. This is the
page you're seeing during the test -- the technical 404 response that says
"I couldn't find a file name 'admin'".

The internals of the server implementation catches Http404 exceptions and
tries a deeper layer; but if a 404 is returned as a view response, it just
gets returned.

So - why does this behaviour change matter?

Well, the default value for MEDIA_ROOT is ''. Which means it matches *all*
URL requests. When DEBUG=False, this doesn't matter - it just means no
files are ever matches by the middle media layer, a 404 is raised, and the
next layer is tried; but when you force DEBUG=True, the media layer
dutifully says "I can't find that file" - for every request it receives. It
doesn't hit this problem for static files because STATIC_ROOT = '/static/'
by default, and the requested URL doesn't match, so the static layer is
never asked to response with file or 404.

You only see this on the LiveTestServer because runserver doesn't include
the Media layer -- it only serve staticfiles and the main app. If have to
manually add the media handler to your URLconf if you want media files to
be served, and that handler will run *after* all the normal URLs have been
tried. And you don't normally see the problem during testing because, as
you've noted, DEBUG is forced to False by the test running infrastructure.

The upside - If you add MEDIA_ROOT='/media/' to settings.py, the problem
goes away, because it gives the media layer something to match against that
won't catch all files.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJxq84_F%3DRrop%2Bn-8i7Krv6zr3Gqdvd%2B%3D%3DK2a1W%3DHfcGJLJiMA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Django ORM support for NoSql databases

[Javier Guerra Giraldez]

On Wed, Dec 18, 2013 at 12:18 PM,   wrote:
>
> Wouldn't an easy (i.e. straightforward) solution be to add an Django "ODM"
> that mirrors the ORM wherever it makes sense?  This sounds pretty close to
> your second solution, except choosing SQL vs NoSQL means users make a more
> explicit choice whether to use the ODM API vs the ORM API.

I think it would be very straightforward to do that at the public API
level, but the ModelForm and the admin use lots of undocumented _meta
fields which would have to be mimicked too.

Still, i'm convinced that this would be the best answer, and a formal
definition of _meta is a big plus, maybe even proportional to the
effort needed.


-- 
Javier

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFkDaoTG_hda%3D05FkQgvFFDzbjshKOjzCii9Mi7UtTw652KFPQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Django ORM support for NoSql databases

[chris . foresman]

On Tuesday, December 17, 2013 8:12:43 PM UTC-6, Russell Keith-Magee wrote:
>
>
> My claim is that complete abstraction of the data store shouldn't be the 
> goal. What we should be aiming for is sufficient API compatibility to allow 
> for two things:
>
>  * ModelForms wrapping a model from a NoSQL data store
>  * An admin representation of a NoSQL data model.
>
> These two functions don't require complex relations. They require 
> relatively simple CRUD operations on a single object; they require an 
> implementation of filter, but not an implementation that allows joins of 
> any kind inside a filter clause. Yes, this will cut off some functionality 
> -- some custom filters, for example, won't be possible. It might be 
> necessary to remove total object counts in admin ListViews -- but then, 
> that would be a good performance boost for PostgreSQL as well. 
>
> But broadly speaking, I see no reason why it shouldn't be possible to put 
> store a data model in NoSQL, and visualise the contents of that model in 
> admin. And you should be able to throw up a Form wrapping that data model. 
>
> I see two ways to achieve this goal. The first is to put an alternate 
> backend into Django's existing ORM. This is the approach attempted by Alex 
> in his GSoC project. It's possible, and actually doesn't require that much 
> work; but there are a couple of unresolved issues, mostly around handling 
> of non-integer automatic primary keys. These are problems that we might be 
> worth addressing anyway, because UUID-based auto primary keys would be a 
> worthwhile extension to SQL data stores.
>
> The second approach is to produce a duck type compatible model API. There 
> aren't (or shouldn't be) any instance checks tied to 
> django.db.models.Model; as long as your NoSQL layer implements the same 
> API, you should be able to drop it into the admin or a ModelForm. Taking 
> this approach would have the side effect that we'd have to clean up the 
> formal definition of _meta, which in itself would be a nice goal.  Added 
> benefit -- this approach doesn't actually require any changes to core 
> itself -- the ducks can be completely external to Django. The only changes 
> to core would be whatever cleanups and documentation the project identifies.
>
> The other, unrelated problem isn't technical at all -- its that that 
> enthusiasm for this problem has pretty much dried up in the core team 
> AFAICT. I certainly don't have any call for using a NoSQL store in my life 
> at the moment. However, if a ready-to-use implementation of either of these 
> approaches were to land on our doorstep, I wouldn't resist them being added 
> to core.
>

Wouldn't an easy (i.e. straightforward) solution be to add an Django "ODM" 
that mirrors the ORM wherever it makes sense?  This sounds pretty close to 
your second solution, except choosing SQL vs NoSQL means users make a more 
explicit choice whether to use the ODM API vs the ORM API.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6f2bcf71-13bd-40cf-bee8-0f5fb264a5c7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Creating a minimal custom user model. Seems last_login is required. Should it be?

[Harry Percival]

Hi all, can't believe I missed this entire thread because googlegroups 
didn't auto-subscribe me to replies.  thanks for the tips.

For the curious, I'm using Mozilla Persona.  Detailed info here: 
http://chimera.labs.oreilly.com/books/123400754/ch14.html

On Monday, 21 October 2013 15:22:07 UTC+1, Xavier Ordoquy wrote:
>
> Hi,
>
> Le 21 oct. 2013 à 16:04, Tino de Bruijn  
> a écrit :
>
> Harry's use case is an interesting one -- his authentication is being done 
>> entirely by an external process, so there's no need for a password field. 
>> Yes, he could just have the password and last_login fields and not use it, 
>> but why should he need to carry around he extra weight when Django doesn't 
>> need it.
>>
>
> @Harry, just out of curiosity, may I ask how you *do* authenticate your 
> users?
>
>
> I can't speak for Harry but using the RemoteUserBackend you don't need the 
> password nor the last_login for Django.
> In my case Apache did the authentication through Kerberos.
> Django's documentation explains more there: 
> https://docs.djangoproject.com/en/dev/howto/auth-remote-user/
>
> Regards,
> Xavier,
> Linovia.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cfc893f6-0177-4e5d-ac30-1ca109e98971%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

LiveServerTestCase, and override_settings(DEBUG=True)

[Harry Percival]

Django's test runner overrides your settings to force DEBUG to be True, 
which I understand the intention behind, but it is occasionally annoying.  
One solution for those cases is to use `override_settings`, but that has 
very weird effects when using `LiveServerTestCase`.

Minimal repro:

django-admin.py startproject myproj
cd myproj
vi tests.py

In tests.py:

from django.test import LiveServerTestCase
from selenium import webdriver
from django.test.utils import override_settings

@override_settings(DEBUG=True)
class MinimalTest(LiveServerTestCase):

def setUp(self):
self.browser = webdriver.Firefox()
self.browser.implicitly_wait(3)

def tearDown(self):
self.browser.quit()


def test_admin_site_is_there(self):
self.browser.get(self.live_server_url + '/admin')
self.assertIn('Django administration', 
self.browser.find_element_by_tag_name('body').text)

Change DEBUG=True to DEBUG=False and this passes.  But, as-is, you get:

  File "/tmp/myproj/test1.py", line 18, in test_admin_site_is_there
self.assertIn('Django administration', 
self.browser.find_element_by_tag_name('body').text)
AssertionError: 'Django administration' not found in 'Page not found 
(404)\nRequest Method: GET\nRequest URL: 
http://localhost:8081/admin\n"admin; does not exist\nYou\'re seeing this 
error because you have DEBUG = True in your Django settings file. Change 
that to False, and Django will display a standard 404 page.'

I'm just using the admin URL because it's one that is activated by 
default.  Normal URLs, eg a home page view, also return 404s.  What's going 
on?



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/eb0d8b58-ea5c-4d7a-9c2a-d302d4adc3ae%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.