Re: Improving MSSQL and Azure SQL support on Django

[Florian Apolloner]

Hi Warren,

> A) django-mssql is viewed as the Phase 1 focus and current preferred 
option with a longer term Phase 2 goal of including MSSQL as a supported 
backend for Django core

Please keep in mind that Phase 2 is something that might never happen. We 
have a tendency to not bloat Django and there is no reason why a database 
backend cannot live outside of core. Tim Graham is currently working on a 
cockroach backend and is running against the builtin Django testsuite (+ 
submitting patches to Django where we need to alter tests due to reliance 
on PKs etc…). This approach works imo very well (I did the same when 
writing a backend for Informix). We are very supportive of 3rd party db 
backends in the sense that we usually quickly address issues in our 
testsuite etc to make testing those external backends against Django 
easier. In that sense there is no strong reason to include the MSSQL 
backend in core. Package management in Python works well enough to allow 
the backend to stay external. 

Personally I think having the django-mssql backend with MS support behind 
it (be that test infra for the existing django-mssql fork etc) is already 
the icing on the cake.

Cheers,
Florian


On Friday, December 4, 2020 at 2:59:43 AM UTC+1 vwa...@gmail.com wrote:

> Thanks for the responses.
>
> @Florian - I've now reached out to the maintainers of ESSolutions as well 
> as the relevant previous Microsoft engagement owners, thanks for the 
> suggestion
>
> @Chris - This is excellent news indeed, we are excited to make progress 
> beginning with some small steps
>
> @r...@whidbey.com - 
> A) django-mssql is viewed as the Phase 1 focus and current preferred 
> option with a longer term Phase 2 goal of including MSSQL as a supported 
> backend for Django core
> B) I've now logged both your pain points as issues in our private repo 
> (which will be made public after internal compliance reviews)
> C) Happy to hear about your scaled production app using Django and MSSQL. 
> Could I ask you to send a "Hello" email to myself at v-wa...@microsoft.com, 
> so that we can follow up with eventual testing?
>
> @dans...@gmail.com - Your security concerns are important and noted, I've 
> flagged this for investigation as a priority
>
> Cheers,
> Warren
> On Friday, 27 November 2020 at 10:57:09 UTC-8 dans...@gmail.com wrote:
>
>> I'd suggest someone talk with professional DBAs for MSSQL.  In my work, 
>> which is Federal government, the DBA told me that disconnecting from PSQL 
>> as "appuser" and attempting to connect to database "postgres" in order to 
>> create the test database violated FISMA.  I had to subclass my own 
>> postgresql backend to create the test user while connected to my actual 
>> database.  I would imagine that the original reasons for doing it this way 
>> harkens back to a time when people used the same database server for 
>> production, staging, qa, and integration, and some of these environments 
>> may not have existed. While in some organizations we have only production, 
>> staging/qa, and development/integration, I would guess there are few cases 
>> where the same database server is used for production and the other 
>> environments.
>>
>> On Fri, Nov 27, 2020 at 1:18 PM r...@whidbey.com  
>> wrote:
>>
>>> Good news.  I've been using Django on MSSQL for about 8 years.  Couple 
>>> of things:
>>> - I've been using pyodbc, not django-mssql.  I note your messages from 
>>> 2015 include it as a library to check for compatibility; what was the 
>>> outcome of that?  Is it proposed that django-mssql become the 
>>> "best-practices" interface for SQL Server?
>>> - Couple of persistent pain points:
>>>   1. Testing.  The Django code that sets up test databases fails with 
>>> MSSQL, while it succeeds with PostGRE, MySQL and SQLite.  The issues seem 
>>> to revolve around setting constraints as the tables are generated, rather 
>>> than holding off and enabling the constraints at the end of the process.
>>>   2. Stored Procedures.  These need to be loaded as an additional step 
>>> in the creation of a database, and don't really have any representation in 
>>> Django per se so migrations, etc don't generally have an idea that they 
>>> exist.
>>>
>>> I'd be happy to test out what you come up with against our system.  It's 
>>> currently serving a custom REST interface with 2-million-plus rows of 
>>> transactions,with clients world-wide, along with a customer-facing web app, 
>>> a staff site and a suite of Tableau reports.
>>>
>>> On Thursday, November 26, 2020 at 5:40:18 PM UTC-8 vwa...@gmail.com 
>>> wrote:
>>>
 Hi All,

 Microsoft has now committed ongoing resources towards improving MSSQL 
 and Azure SQL support for Django. We're currently focused on internal 
 compliance and forking the ESSolutions django-mssql-backend 
 , adding testing 
 pipelines, refactoring the Django DB engine naming convention, and 
 

Re: Improving MSSQL and Azure SQL support on Django

[Warren Chu]

Thanks for the responses.

@Florian - I've now reached out to the maintainers of ESSolutions as well 
as the relevant previous Microsoft engagement owners, thanks for the 
suggestion

@Chris - This is excellent news indeed, we are excited to make progress 
beginning with some small steps

@r...@whidbey.com - 
A) django-mssql is viewed as the Phase 1 focus and current preferred option 
with a longer term Phase 2 goal of including MSSQL as a supported backend 
for Django core
B) I've now logged both your pain points as issues in our private repo 
(which will be made public after internal compliance reviews)
C) Happy to hear about your scaled production app using Django and MSSQL. 
Could I ask you to send a "Hello" email to myself at 
v-war...@microsoft.com, so that we can follow up with eventual testing?

@dans...@gmail.com - Your security concerns are important and noted, I've 
flagged this for investigation as a priority

Cheers,
Warren
On Friday, 27 November 2020 at 10:57:09 UTC-8 dans...@gmail.com wrote:

> I'd suggest someone talk with professional DBAs for MSSQL.  In my work, 
> which is Federal government, the DBA told me that disconnecting from PSQL 
> as "appuser" and attempting to connect to database "postgres" in order to 
> create the test database violated FISMA.  I had to subclass my own 
> postgresql backend to create the test user while connected to my actual 
> database.  I would imagine that the original reasons for doing it this way 
> harkens back to a time when people used the same database server for 
> production, staging, qa, and integration, and some of these environments 
> may not have existed. While in some organizations we have only production, 
> staging/qa, and development/integration, I would guess there are few cases 
> where the same database server is used for production and the other 
> environments.
>
> On Fri, Nov 27, 2020 at 1:18 PM r...@whidbey.com  wrote:
>
>> Good news.  I've been using Django on MSSQL for about 8 years.  Couple of 
>> things:
>> - I've been using pyodbc, not django-mssql.  I note your messages from 
>> 2015 include it as a library to check for compatibility; what was the 
>> outcome of that?  Is it proposed that django-mssql become the 
>> "best-practices" interface for SQL Server?
>> - Couple of persistent pain points:
>>   1. Testing.  The Django code that sets up test databases fails with 
>> MSSQL, while it succeeds with PostGRE, MySQL and SQLite.  The issues seem 
>> to revolve around setting constraints as the tables are generated, rather 
>> than holding off and enabling the constraints at the end of the process.
>>   2. Stored Procedures.  These need to be loaded as an additional step in 
>> the creation of a database, and don't really have any representation in 
>> Django per se so migrations, etc don't generally have an idea that they 
>> exist.
>>
>> I'd be happy to test out what you come up with against our system.  It's 
>> currently serving a custom REST interface with 2-million-plus rows of 
>> transactions,with clients world-wide, along with a customer-facing web app, 
>> a staff site and a suite of Tableau reports.
>>
>> On Thursday, November 26, 2020 at 5:40:18 PM UTC-8 vwa...@gmail.com 
>> wrote:
>>
>>> Hi All,
>>>
>>> Microsoft has now committed ongoing resources towards improving MSSQL 
>>> and Azure SQL support for Django. We're currently focused on internal 
>>> compliance and forking the ESSolutions django-mssql-backend 
>>> , adding testing 
>>> pipelines, refactoring the Django DB engine naming convention, and 
>>> addressing current test suite errors.
>>>
>>> We'd love to hear from current mssql-backend maintainers as well as 
>>> mssql-backend users about the existing issues and feature requests that we 
>>> should be prioritizing.
>>>
>>> We looking forward to engaging the community and working towards MSSQL 
>>> as a first-class supported backend for Django.
>>>
>>> -Warren
>>>
>>> On Wednesday, 4 December 2019 at 07:05:25 UTC-8 Tim Allen wrote:
>>>
 Hi Sean, just an update from what I know.

 We are still waiting for a reply from Microsoft. They're a large 
 company, so understandably, it takes a little while.


 For now, if people need to get onto Django 2.2 for long term support 
 (which will last until April, 2022), you can use this package:

 https://github.com/ESSolutions/django-mssql-backend I've been running 
 it in production for months without incident. Of course, YMMV.


 If Microsoft and/or the DSF end up wanting to bring support under the 
 Django umbrella, the django-mssql-backend repository is a possible 
 starting point, IMHO.

 The django-mssql-backend is currently being developed and support for 
 Django 3.0 is being worked on: ESSolutions/django-mssql-backend#18 
 


 Regards,


 Tim

Re: `select_related` relation control

[Roger Gammans]

On Thu, 2020-12-03 at 11:58 +, Adam Johnson wrote:
> What prevents you from using select_related with an explicit list of
> relations that you *do* want? 

well the fact the select_related() call is in third party code. In the
most common case for us  this is contrib.admin.

So we are forced to be a bit more explicit in the ModelAdmin Meta
definition than we would normally be. It no
bad thing., but it occasionally catches team members out. It fails fast
here, so it's not going to go unnoticed like
a performance problem can. 

Thanks for the blog link and the  reference to django-auto-prefetch,
there is some good stuff in there to think about.


-- 
Roger Gammans 
Gamma Science Ltd. (GB Nr. 07356014 )

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7913088a6a1f28e325c57c0ab2892a3b6cbb2576.camel%40gammascience.co.uk.

Re: `select_related` relation control

[Adam Johnson]

What prevents you from using select_related with an explicit list of
relations that you *do* want? That's more maintainable as it prevents
overfetching when you add new relations. If it were up to me I'd remove the
"select_related()-selects-all-relations-behaviour" - it's too implicit and
can lead to massive queries without much thought.

In many situations prefetch_related() outperforms select_related() as it
doesn't pull back repeat copies of low-cardinality related objects. I
covered this a bit in my N+1 blog post here:
https://adamj.eu/tech/2020/09/01/django-and-the-n-plus-one-queries-problem/
.

I'd also point you to django-auto-prefetch which performs prefetches across
a queryset's objects on first access, eliminating nearly 90%+ of the need
to manually tune select or prefetch_related calls. That's also covered in
the blog post.


On Thu, 3 Dec 2020 at 10:24, Roger Gammans 
wrote:

> Hi
>
> A recent question on one of the other django mailing lists reminded me of
> this: I have a very edge (and almost certainly off the end of 'supported')
> use case, where I would like to mark some relations on a model, as having
> select_related() disabled.
>
> More specifically the requirement is that qs.select_related() (with no
> args) doesn't include the marked relations. It doesn't look like there is a
> way of doing this in the ORM as it stands - is this correct?
>
> I could probably work up this feature as a field flag, but given that it's
> completely niche, I wondered what the chance of the core team accepting it
> was. (with unittests)?
>
>
> --
>
> Roger Gammans 
> Gamma Science Ltd. (GB Nr. 07356014 )
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/0d9b15f264065bf090e7413e80f27d022d21a227.camel%40gammascience.co.uk
> 
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM3bJz-QB79LTjsXUJo7_K5PEopvYSAy9j1YE%3DvETTn9rQ%40mail.gmail.com.

`select_related` relation control

[Roger Gammans]

Hi 

A recent question on one of the other django mailing lists reminded me
of this: I have a very edge (and almost certainly off the end of
'supported') use case, where I would like to mark some relations on a
model, as having select_related() disabled.

More specifically the requirement is that qs.select_related()  (with no
args) doesn't include the marked relations. It doesn't look like there
is a way of doing this in the ORM as it stands - is this correct?

I could probably work up this feature as a field flag, but given that
it's completely niche, I wondered what the chance of the core team
accepting it was. (with unittests)?



-- 
Roger Gammans 
Gamma Science Ltd. (GB Nr. 07356014 )

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0d9b15f264065bf090e7413e80f27d022d21a227.camel%40gammascience.co.uk.

Re: In reference to Ticket 15578

[Vineet Sharma]

Hi Muskan,
I am Vineet, and I am also new in here for contributing. I wanted to ask 
were you able to resolve the ticket mentioned above?
I wanted some help in resolving a ticket of mine, I hope you could help me 
out.

Thanks & Regards

Vineet Sharma

On Sunday, November 29, 2020 at 8:54:45 PM UTC+5:30 muskan...@gmail.com 
wrote:

> This is in reference to ticket #15578 
> . 
> Since I'm very new here, I am nervous to do anything by myself.
> Trying to start small, I want to take up this documentation issue. Do let 
> me know if What i suggested in comment 9 is an improvement. 
> If what I suggested is in fact correct, I'd like to assign myself the 
> ticket and fix this, if that's okay. If its incorrect, do tell me where I 
> can improve, I'd still like to fix it..
>
> Regards
> Muskan Vaswan
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a2bebf4a-ccb3-4899-a67e-563f63e46e4cn%40googlegroups.com.