Re: [ GSOC 2016 ] providing standard interface for NoSQL databases

[girish ramnani]

In the beginning I thought about making an api which supports column 
databases such as cassendra but would mean that i would need to first find 
commonalities between all 3 ( column , document and relational ) but such 
commonalities would be lesser and so finally i would require to create a 
separate api instead of trying to provide it with db.Model.

So i will be target only document based databases like mongo , rethink etc.

There is an opinion I seek from the community.

1 ) Providing a separate Model written specifically for document database 
would provide easier addition of new features related to real time 
capabilities which could be used after ASGI gets merged. Also a developer 
usually knows before writing any code which type of database (relational or 
non-relational ) so in my opinion there wouldn't be developers who would 
use sqlite in dev mode and then change it to mongo in production. So would 
it be better to provide a separate Model for document databases ?

2 ) On the other hand , using the db.Model will be easier to use as the api 
is already familiar with the developers but using the same api suggests a 
certain level of similarity between databases which cant be seen in case of 
relational and document databases. 
  
So would the community be open to adding a seperate Model class eg. 
DocumentModel or Collection .

There are alot of problems which are to be address ( some mentioned by Anssi 
Kääriäinen ).

The multiple database support,Forms,migrations,Meta,django admin , django 
auth etc. I would try to address the problems in my proposal as in-depth as 
possible. 


On Wednesday, March 16, 2016 at 3:20:25 AM UTC+5:30, Curtis Maloney wrote:
>
> It sounds like by "NoSQL DBS" you mean specifically "Document store 
> DBMS".  Is this correct? 
>
> (I'm sure most people know my rant about how "NoSQL" is a misnomer, and 
> it's really "Non-relational" that people mean - it's not SQL per se 
> they're avoiding, it's the relational model) 
>
> The NoSQL/Non-rel name is really too vague, as it covers "anything 
> that's not SQL / Relational"... which can cover key/value, document, 
> graph, log, and even filesystem or email, when it comes down to it. 
>
> I think it will help if you clarify what model you _do_ target, not 
> which you do _not_. 
>
> All that said/vented, I agree it would be nice to have a common 
> abstraction layer atop various document stores.  The trick now will be 
> finding sufficient commonalities. 
>
> I feel that the ORM's relation-span/transforms/lookups syntax would be 
> equally viable on nested documents. 
>
> One of the benefits of an abstraction layer is allowing other tools to 
> generically interact with these new objects.  What plans do you have for 
> how to support non-flat documents in, for instance, Forms? 
>
>
> -- 
> Curtis 
>
> On 16/03/16 05:29, girish ramnani wrote: 
> > 
> > Hello, 
> > 
> > I would like to propose providing a standard interface for NoSQL 
> > databases . Some the points to consider are: 
> > 
> >  1. The /Fields /which are used to define the schema of the sql 
> > databases can also be used to provide in No sql databases. 
> >  2. In NoSQL dbs ,developers usually store data in denormalized form. so 
> > in two related collections a meta tag /denormal = True /can be 
> > provided. Which would make that collection a child of the other 
> > model ( related ). In normal scenario the collections will be 
> > connected using _id used as a foreign key. 
> >  3. providing contrib.mongo with Field such as GridFSField for providing 
> > support for GridFS 
> >  4. considering the database transaction transaction blocking ( PyMongo 
> > provides blocking api for mongo ) 
> >  5. creating Mixins such as /FlexibleMixin /which would allow Models to 
> > have dynamic fields. i.e directly insert fields which are not listed 
> > in the Model schema. 
> > 
> > 
> > there are lot more things (Migrations, settings etc) to consider but 
> > before that needed to know that I am going in the right direction or 
> > not?. Please suggest. 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "Django developers (Contributions to Django itself)" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> > an email to django-develop...@googlegroups.com  
> > . 
> > To post to this group, send email to django-d...@googlegroups.com 
>  
> > . 
> > Visit this group at https://groups.google.com/group/django-developers. 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com?utm_medium=email_source=footer>.
>  
>
> > For more options, visit 

Re: [ GSOC 2016 ] providing standard interface for NoSQL databases

[Anssi Kääriäinen]

There are two goals when it comes to document DB support. First goal
is to allow usage of models written specifically for document oriented
databases. This is something that should be possible to achieve even
if there are some hard problems to solve. The other goal is to allow
usage of any Django model with document oriented databases just like
you would use it with relational databases. This goal is extremely
hard to achieve.

The reason the latter one is hard is that for example ForeignKey and
ManyToManyField do not make much sense with document oriented
databases. In addition most Django models use auto-incrementing
primary keys, and pretty much no distributed database supports those
properly. So, if you take a random Django model (say
django.contrib.auth.models.Permission) it will not be easy to fit that
model into document oriented databases. If I recall correctly, one of
the big problems for the GSoC project Russell mentioned was support
for any Django model with document oriented databases.

I believe there would be lots of uses for models which are written
specifically for document oriented databases. So +1 to the general
idea.

 - Anssi


On Wed, Mar 16, 2016 at 3:10 AM, Russell Keith-Magee
 wrote:
>
> It would also be worthwhile looking into the prior art on this topic. We had
> a GSoC project for non-relational data stores a few years back, and came to
> the conclusion that it probably wasn’t viable. You can dig into the archives
> to find out why.
>
> If you’re interested in adding non-relational support to Django, my
> inclination is that you’ll get more traction proposing a specific backend -
> I outlined what would be involved in my DjangoCon US talk last year:
>
> https://www.youtube.com/watch?v=VgM0qmpHDiE
>
> Yours,
> Russ Magee %-)
>
> On Wed, Mar 16, 2016 at 5:50 AM, Curtis Maloney  wrote:
>>
>> It sounds like by "NoSQL DBS" you mean specifically "Document store DBMS".
>> Is this correct?
>>
>> (I'm sure most people know my rant about how "NoSQL" is a misnomer, and
>> it's really "Non-relational" that people mean - it's not SQL per se they're
>> avoiding, it's the relational model)
>>
>> The NoSQL/Non-rel name is really too vague, as it covers "anything that's
>> not SQL / Relational"... which can cover key/value, document, graph, log,
>> and even filesystem or email, when it comes down to it.
>>
>> I think it will help if you clarify what model you _do_ target, not which
>> you do _not_.
>>
>> All that said/vented, I agree it would be nice to have a common
>> abstraction layer atop various document stores.  The trick now will be
>> finding sufficient commonalities.
>>
>> I feel that the ORM's relation-span/transforms/lookups syntax would be
>> equally viable on nested documents.
>>
>> One of the benefits of an abstraction layer is allowing other tools to
>> generically interact with these new objects.  What plans do you have for how
>> to support non-flat documents in, for instance, Forms?
>>
>>
>> --
>> Curtis
>>
>> On 16/03/16 05:29, girish ramnani wrote:
>>>
>>>
>>> Hello,
>>>
>>> I would like to propose providing a standard interface for NoSQL
>>> databases . Some the points to consider are:
>>>
>>>  1. The /Fields /which are used to define the schema of the sql
>>> databases can also be used to provide in No sql databases.
>>>  2. In NoSQL dbs ,developers usually store data in denormalized form. so
>>> in two related collections a meta tag /denormal = True /can be
>>> provided. Which would make that collection a child of the other
>>> model ( related ). In normal scenario the collections will be
>>> connected using _id used as a foreign key.
>>>  3. providing contrib.mongo with Field such as GridFSField for providing
>>> support for GridFS
>>>  4. considering the database transaction transaction blocking ( PyMongo
>>> provides blocking api for mongo )
>>>  5. creating Mixins such as /FlexibleMixin /which would allow Models to
>>> have dynamic fields. i.e directly insert fields which are not listed
>>> in the Model schema.
>>>
>>>
>>> there are lot more things (Migrations, settings etc) to consider but
>>> before that needed to know that I am going in the right direction or
>>> not?. Please suggest.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-developers+unsubscr...@googlegroups.com
>>> .
>>> To post to this group, send email to django-developers@googlegroups.com
>>> .
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>>
>>> https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com

Re: [ GSOC 2016 ] providing standard interface for NoSQL databases

[Russell Keith-Magee]

It would also be worthwhile looking into the prior art on this topic. We
had a GSoC project for non-relational data stores a few years back, and
came to the conclusion that it probably wasn’t viable. You can dig into the
archives to find out why.

If you’re interested in adding non-relational support to Django, my
inclination is that you’ll get more traction proposing a specific backend -
I outlined what would be involved in my DjangoCon US talk last year:

https://www.youtube.com/watch?v=VgM0qmpHDiE

Yours,
Russ Magee %-)

On Wed, Mar 16, 2016 at 5:50 AM, Curtis Maloney  wrote:

> It sounds like by "NoSQL DBS" you mean specifically "Document store
> DBMS".  Is this correct?
>
> (I'm sure most people know my rant about how "NoSQL" is a misnomer, and
> it's really "Non-relational" that people mean - it's not SQL per se they're
> avoiding, it's the relational model)
>
> The NoSQL/Non-rel name is really too vague, as it covers "anything that's
> not SQL / Relational"... which can cover key/value, document, graph, log,
> and even filesystem or email, when it comes down to it.
>
> I think it will help if you clarify what model you _do_ target, not which
> you do _not_.
>
> All that said/vented, I agree it would be nice to have a common
> abstraction layer atop various document stores.  The trick now will be
> finding sufficient commonalities.
>
> I feel that the ORM's relation-span/transforms/lookups syntax would be
> equally viable on nested documents.
>
> One of the benefits of an abstraction layer is allowing other tools to
> generically interact with these new objects.  What plans do you have for
> how to support non-flat documents in, for instance, Forms?
>
>
> --
> Curtis
>
> On 16/03/16 05:29, girish ramnani wrote:
>
>>
>> Hello,
>>
>> I would like to propose providing a standard interface for NoSQL
>> databases . Some the points to consider are:
>>
>>  1. The /Fields /which are used to define the schema of the sql
>> databases can also be used to provide in No sql databases.
>>  2. In NoSQL dbs ,developers usually store data in denormalized form. so
>> in two related collections a meta tag /denormal = True /can be
>> provided. Which would make that collection a child of the other
>> model ( related ). In normal scenario the collections will be
>> connected using _id used as a foreign key.
>>  3. providing contrib.mongo with Field such as GridFSField for providing
>> support for GridFS
>>  4. considering the database transaction transaction blocking ( PyMongo
>> provides blocking api for mongo )
>>  5. creating Mixins such as /FlexibleMixin /which would allow Models to
>> have dynamic fields. i.e directly insert fields which are not listed
>> in the Model schema.
>>
>>
>> there are lot more things (Migrations, settings etc) to consider but
>> before that needed to know that I am going in the right direction or
>> not?. Please suggest.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to django-developers+unsubscr...@googlegroups.com
>> .
>> To post to this group, send email to django-developers@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>>
>> https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com
>> <
>> https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com?utm_medium=email_source=footer
>> >.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/56E88390.40005%40tinbrain.net
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 

Re: [ GSOC 2016 ] providing standard interface for NoSQL databases

[Curtis Maloney]

It sounds like by "NoSQL DBS" you mean specifically "Document store 
DBMS".  Is this correct?


(I'm sure most people know my rant about how "NoSQL" is a misnomer, and 
it's really "Non-relational" that people mean - it's not SQL per se 
they're avoiding, it's the relational model)


The NoSQL/Non-rel name is really too vague, as it covers "anything 
that's not SQL / Relational"... which can cover key/value, document, 
graph, log, and even filesystem or email, when it comes down to it.


I think it will help if you clarify what model you _do_ target, not 
which you do _not_.


All that said/vented, I agree it would be nice to have a common 
abstraction layer atop various document stores.  The trick now will be 
finding sufficient commonalities.


I feel that the ORM's relation-span/transforms/lookups syntax would be 
equally viable on nested documents.


One of the benefits of an abstraction layer is allowing other tools to 
generically interact with these new objects.  What plans do you have for 
how to support non-flat documents in, for instance, Forms?



--
Curtis

On 16/03/16 05:29, girish ramnani wrote:


Hello,

I would like to propose providing a standard interface for NoSQL
databases . Some the points to consider are:

 1. The /Fields /which are used to define the schema of the sql
databases can also be used to provide in No sql databases.
 2. In NoSQL dbs ,developers usually store data in denormalized form. so
in two related collections a meta tag /denormal = True /can be
provided. Which would make that collection a child of the other
model ( related ). In normal scenario the collections will be
connected using _id used as a foreign key.
 3. providing contrib.mongo with Field such as GridFSField for providing
support for GridFS
 4. considering the database transaction transaction blocking ( PyMongo
provides blocking api for mongo )
 5. creating Mixins such as /FlexibleMixin /which would allow Models to
have dynamic fields. i.e directly insert fields which are not listed
in the Model schema.


there are lot more things (Migrations, settings etc) to consider but
before that needed to know that I am going in the right direction or
not?. Please suggest.

--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to django-developers+unsubscr...@googlegroups.com
.
To post to this group, send email to django-developers@googlegroups.com
.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/56E88390.40005%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

[ GSOC 2016 ] providing standard interface for NoSQL databases

[girish ramnani]

Hello,

I would like to propose providing a standard interface for NoSQL databases 
. Some the points to consider are:

   1. The *Fields *which are used to define the schema of the sql databases 
   can also be used to provide in No sql databases.
   2. In NoSQL dbs ,developers usually store data in denormalized form. so 
   in two related collections a meta tag *denormal = True *can be provided. 
   Which would make that collection a child of the other model ( related ). In 
   normal scenario the collections will be connected using _id used as a 
   foreign key.
   3. providing contrib.mongo with Field such as GridFSField for providing 
   support for GridFS
   4. considering the database transaction transaction blocking ( PyMongo 
   provides blocking api for mongo )
   5. creating Mixins such as *FlexibleMixin *which would allow Models to 
   have dynamic fields. i.e directly insert fields which are not listed in the 
   Model schema.


there are lot more things (Migrations, settings etc) to consider but before 
that needed to know that I am going in the right direction or not?. Please 
suggest.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.