Re: Withdrawing Channels from 1.10

[Mark Lavin]

Yes thank you Andrew for your continued work to move this conversation 
forward. I hope that Channels can continue to grow as an external package 
under the Django umbrella and bring on more contributors and improvements.

Best,

Mark

On Tuesday, May 10, 2016 at 12:44:21 PM UTC-4, Ryan Hiebert wrote:
>
> Thank you, Andrew, for your hard work. Channels is an exciting new 
> feature, and I'm glad that you're bringing it together. You're doing an 
> excellent job.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c4fb72c3-0c41-472b-bdd2-2d462333e046%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: My Take on Django Channels

[Mark Lavin]

Ryan,

Sorry if you felt I was ignoring your reply to focus on the discussion with 
Andrew. You both made a lot of the same points at about the same time but I 
did want to touch on a couple things.

On Thursday, May 5, 2016 at 4:21:59 PM UTC-4, Ryan Hiebert wrote:
>
> Thank you, Mark, for starting this discussion. I, too, found myself simply 
> accepting that channels was the right way to go, despite having the same 
> questions you do. I realize this shouldn't be, so I've chimed in on some of 
> your comments. 
>
> > On May 5, 2016, at 2:34 PM, Mark Lavin > 
> wrote: 
> > 
> > [snip] 
> > 
> > The Channel API is built more around a simple queue/list rather than a 
> full messaging layer. [snip] Kombu supports  [snip]. 
>
> The API was purposefully limited, because channels shouldn't need all 
> those capabilities. All this is spelled out in the documentation, which I 
> know you already understand because you've mentioned it elsewhere. I think 
> that the choice to use a more limited API makes sense, though that doesn't 
> necessarily mean that it is the right choice. 
> > 
> > [snip description of architecture] 
>
> First off, the concerns you mention make a lot of sense to me, and I've 
> been thinking along the same lines. 
>
> I've been considering if having an alternative to Daphne that only used 
> channels for websockets, but used WSGI for everything else. Or some 
> alternative split where some requests would be ASGI and some WSGI. I've 
> tested a bit the latency overhead that using channels adds (on my local 
> machine even), and it's not insignificant. I agree that finding a solution 
> that doesn't so drastically slow down the requests that we've already 
> worked hard to optimize is important. I'm not yet sure the right way to do 
> that. 
>
> As far as scaling, it is apparent to me that it will be very important to 
> have the workers split out, in a similar way to how we have different 
> celery instances processing different queues. This allows us to scale those 
> queues separately. While it doesn't appear to exist in the current 
> implementation, the channel names are obviously suited to such a split, and 
> I'd expect channels to grow the feature of selecting which channels a 
> worker should be processing (forgive me if I've just missed this 
> capability, Andrew). 
>

Similar to Celery, the workers can listen on only certain channels or 
exclude listening on channels which is sort of a means of doing 
priority https://github.com/andrewgodwin/channels/issues/116 I would also 
like to see this expanded or more have the use case more clearly documented.
 

> > 
> > [[ comments on how this makes deployment harder ]] 
>
> ASGI is definitely more complex that WSGI. It's this complexity that gives 
> it power. However, to the best of my knowledge, there's not a push to be 
> dropping WSGI. If you're doing a simple request/response site, then you 
> don't need the complexity, and you probably should be using WSGI. However, 
> if you need it, having ASGI standardized in Django will help the community 
> build on the power that it brings. 

> 
> > Channels claims to have a better zero-downtime deployment story. 
> However, in practice I’m not convinced that will be true. [snip] 
>
> I've been concerned about this as well. On Heroku my web dynos don't go 
> down, because the new ones are booted up while the old ones are running, 
> and then a switch is flipped to have the router use the new dynos. Worker 
> dynos, however, do get shut down. Daphne won't be enough to keep my site 
> functioning. This is another reason I was thinking of a hybrid WSGI/ASGI 
> server. 
> > 
> > There is an idea floating around of using Channels for background 
> jobs/Celery replacement. It is not/should not be. [snip reasons] 
>
> It's not a Celery replacement. However, this simple interface may be good 
> enough for many things. Anything that doesn't use celery's `acks_late` is a 
> candidate, because in those cases even Celery doesn't guarantee delivery, 
> and ASGI is a simpler interface than the powerful, glorious behemoth that 
> is Celery.  


This isn't the place for a long discussion about the inner workings of 
Celery but I don't believe this is true. The prefetched tasks are not 
acknowledged until they are delivered to a worker for processing. Once 
delivered, the worker might die/be killed before it can complete the task 
but the message was delivered. That's the gap that acks_late solves: 
between the message delivery and the completion of the task. Not all 
brokers support message acknowledgement natively an

Re: My Take on Django Channels

[Mark Lavin]

Yes I agree that we do want different things and have different goals. 
There is nothing wrong with coming to a state of respectful disagreement. 
I'm glad that some of the feedback could be helpful and I hope it can be 
incorporated into Channels.

As for a DEP, that would be nice and I'd love to participate in that 
process. To this point I don't feel like the argument for Channels has been 
weighed against existing alternative approaches which is largely what I've 
tried to start here. I mention the DEP process as a source of my own 
resentment for this change and part of the reason I've held this feedback 
in for so long. Again I don't think that was fair to you or the Django 
community to do so. You've been open about your work and your goals. I had 
plenty of opportunity to voice my concern to you publicly or privately and 
I chose not to do so for arguably petty reasons. I don't want to see this 
work blocked because of a lack of DEP if it has the support of the core 
team and the larger community. I've said my piece about this work and I'm 
letting those past feelings go so that I can contribute more constructively 
to this conversation.

- Mark

On Thursday, May 5, 2016 at 8:52:17 PM UTC-4, Andrew Godwin wrote:
>
>
>
> On Thu, May 5, 2016 at 5:13 PM, Mark Lavin  > wrote:
>
>> Yes I agree with the value of a standardized way of communicating between 
>> these processes and I listed that as a highlight of Channels, though it 
>> quickly shifted into criticism. I think that's where we are crossing paths 
>> with relation to Kombu/AMQP as well. I find the messaging aspect of 
>> Channels far more interesting and valuable than ASGI as a larger 
>> specification. Messaging I do think needs to be network transparent. I just 
>> don't like that aspect tied into the HTTP handling. At this point I'm not 
>> sure how to decouple the messaging aspect from the HTTP layer since I feel 
>> they are very tightly bound in ASGI.
>>
>
> I see what you mean; HTTP is definitely less of a fit to ASGI than 
> WebSockets, and it wasn't even in there at all initially, but I felt that 
> the ability to unify everything inside Django to be a consumer was too 
> strong to pass up (plus the fact that it allowed long-polling HTTP which I 
> still use a lot in lieu of WebSocket support, mostly for work reasons).
>  
>
>>
>> Honestly I don't think Django *needs* tightly integrated websocket 
>> support but I do see the value in it so we aren't at a complete impasse. I 
>> suppose that's why it's my general preference to see a third-party solution 
>> gain traction before it's included. I played with integrating Django + 
>> aiohttp a few months ago. Nothing serious and I wouldn't call it an 
>> alternate proposal. It's barely a proof of concept: 
>> https://github.com/mlavin/aiodjango. My general inclination is that 
>> (insert wild hand waving) 
>> django.contrib.aiohttp/django.contrib.twisted/django.contrib.tornado would 
>> be the way forward for Django + websockets without a full scale rewrite of 
>> the WSGI specification.
>>
>>
> The other track for this was definitely to go the South route and have it 
> run externally, but based on my previous experience with that route it is 
> not scalable from a people perspective.
>
> I personally see this as something where any single third-party solution 
> is not going to gain enough traction to be tested and tried enough unless 
> it's defacto recommended by Django itself, at which point it's close to 
> being a core module with provisional status.
>
> I feel like we're never going to quite agree on the approach here; I've 
> explained my stance, you have explained yours, and I think we both have a 
> good idea where we stand. I agree with some of your concerns, especially 
> around introducing more moving parts, but then modern websites have so many 
> already my concerns are perpetually high.
>
> Given your feedback, I do want to work on a local, cross-process ASGI 
> backend and write up a full deployment story that uses WSGI servers for 
> HTTP and Daphne+worker servers for WebSockets, and have it as a top example 
> for what larger sites should do to deploy WebSockets initially; I think 
> that's an important piece of communication to show that this is only as 
> opt-in as you want it to be.
>
> I'm also hopeful that the introduction of chat, email and other protocol 
> (e.g. IoT) interface servers to further highlight the flexibility of a 
> general messaging + worker system will help move us towards a future with 
> less moving parts; ASGI and Channels was always meant to be 

Re: My Take on Django Channels

[Mark Lavin]

Yes I agree with the value of a standardized way of communicating between 
these processes and I listed that as a highlight of Channels, though it 
quickly shifted into criticism. I think that's where we are crossing paths 
with relation to Kombu/AMQP as well. I find the messaging aspect of 
Channels far more interesting and valuable than ASGI as a larger 
specification. Messaging I do think needs to be network transparent. I just 
don't like that aspect tied into the HTTP handling. At this point I'm not 
sure how to decouple the messaging aspect from the HTTP layer since I feel 
they are very tightly bound in ASGI.

Honestly I don't think Django *needs* tightly integrated websocket support 
but I do see the value in it so we aren't at a complete impasse. I suppose 
that's why it's my general preference to see a third-party solution gain 
traction before it's included. I played with integrating Django + aiohttp a 
few months ago. Nothing serious and I wouldn't call it an alternate 
proposal. It's barely a proof of 
concept: https://github.com/mlavin/aiodjango. My general inclination is 
that (insert wild hand waving) 
django.contrib.aiohttp/django.contrib.twisted/django.contrib.tornado would 
be the way forward for Django + websockets without a full scale rewrite of 
the WSGI specification.

Not sure if I touched on all of your questions so please let me know if it 
seems like I'm skipping over something.

- Mark

On Thursday, May 5, 2016 at 6:31:05 PM UTC-4, Andrew Godwin wrote:
>
>
>
> On Thu, May 5, 2016 at 2:19 PM, Mark Lavin  > wrote:
>
>> Thank you for your comments and I have some brief replies.
>>
>>
>> If I'm understanding it correctly, groups are an emulated broadcast. I'm 
>> saying it would be an advantage for it to use pub/sub but it does not.
>>
>
> You are correct; the reason Redis pub/sub is not used is because the ASGI 
> API allows applications to not listen continuously on channels and instead 
> check in every so often, so it uses lists so there's some persistence; this 
> could be changed, though. I do want to improve the group send function so 
> it runs on Lua inside Redis rather than multi-sending from outside, however.
>  
>
>>  
>>
>>>
>>> I've always tried to be clear that it is not a Celery replacement but 
>>> instead a way to offload some non-critical task if required.
>>>
>>
>> I don't agree that this has been clear. That is my primary criticism 
>> here. I don't think this should be encouraged. Ryan's reply continues with 
>> this confusion.
>>
>
> I would love to work with you on clearing this up, then; trying to 
> communicate what the design is intended to be is one of the hardest parts 
> of this project, especially considering there are so many avenues people 
> hear about this stuff through (and the fact that I do think _some_ 
> non-critical tasks could be offloaded into channels consumers, just not the 
> sort Celery is currently used for).
>  
>
>>
>> Yes the lock-in is an exaggeration, however, given the poor 
>> support/upkeep for third-party DB backends, I doubt the community will have 
>> better luck with Channel backends not officially supported by the Django 
>> core team. I'd be happy to be wrong here.
>>
>
> Yes, that's a fair comparison. There was even an effort to try and get a 
> second one going and ready to use before merge but unfortunately it didn't 
> get anywhere yet.
>  
>
>>
>> Kombu is not to be confused with Celery. Kombu is a general purpose 
>> AMQP/messaging abstraction library. I don't think we agree on its potential 
>> role here. Perhaps it's better stated that I think Channel's minimalist API 
>> is too minimalist. I would prefer if additional AMQP-like abstractions 
>> existed such as topic routing and QoS.
>>
>
> I understand what Kombu is (though it's maintained by the Celery team from 
> what I understand, which is why I refer to them collectively). I still 
> maintain that the design of AMQP and Kombu is unsuited for what I am trying 
> to accomplish here; maybe what I am trying to accomplish is wrong, and I'm 
> happy to argue that point, but based on what I'm trying to do, AMQP and 
> similar abstractions are not a good fit - and I did write one of the 
> earlier versions of Channels on top of Celery as an experiment.
>  
>
>>
>>> ASGI is essentially meant to be an implementation of the CSP/Go style of 
>>> message-passing interprocess communication, but cross-network rather than 
>>> merely cross-thread or cross-process as I believe that network transparenc

Re: My Take on Django Channels

[Mark Lavin]

Thank you for your comments and I have some brief replies.

On Thursday, May 5, 2016 at 4:20:06 PM UTC-4, Andrew Godwin wrote:
>
>
>
> On Thu, May 5, 2016 at 12:34 PM, Mark Lavin  > wrote:
>
> The main gains are (in my opinion):
>  - The same server process can serve both HTTP and WebSockets without path 
> prefixing (auto-negotiation based on the Upgrade header); without this you 
> need an extra web layer in front to route requests to the right backend 
> server
>  - HTTP long-polling is supported via the same mechanism (like WebSockets, 
> it does not fit inside the WSGI paradigm in a performant way)
>  - You get to run less processes overall
>

As noted I don't see serving them both as an advantage. Also given that 
daphne is single-thread/eventloop based, it's likely that frontend proxy 
already would be needed to handle balance multiple processes or SSL 
termination. I don't think this reduces the number of processes. As stated 
I think it's the same.
 

>
> Firstly, nothing in channels uses pub/sub - channels deliver to a single 
> reader of a queue, and thus cannot be built on a broadcast solution like 
> pub/sub.  
>
 
>

If I'm understanding it correctly, groups are an emulated broadcast. I'm 
saying it would be an advantage for it to use pub/sub but it does not.
 

>
> I've always tried to be clear that it is not a Celery replacement but 
> instead a way to offload some non-critical task if required.
>

I don't agree that this has been clear. That is my primary criticism here. 
I don't think this should be encouraged. Ryan's reply continues with this 
confusion.
 

>
>> So Channels is at best on par with the existing available approaches and 
>> at worst adds a bunch of latency, potentially dropped messages, and new 
>> points of failure while taking up more resources and locks everyone into 
>> using Redis. It does provide a clear message framework but in my opinion 
>> it’s too naive to be useful. Given the complexity in the space I don’t 
>> trust anything built from the ground up without having a meaningful 
>> production deployment to prove it out. It has taken Kombu many years to 
>> mature and I don’t think it can be rewritten easily.
>>
>
> a) ASGI does not lock everyone into using Redis; it just so happens that 
> is the first backend I have written. It is designed to run against other 
> suitable datastores or socket protocols and we have the money to fund such 
> an endeavour.
>
> b) Kombu solves a different problem - that of abstracting task queues - 
> and it would still be my first choice for that; I have used it for many 
> years and it would continue to be my choice for task queuing.
>

Yes the lock-in is an exaggeration, however, given the poor support/upkeep 
for third-party DB backends, I doubt the community will have better luck 
with Channel backends not officially supported by the Django core team. I'd 
be happy to be wrong here.

Kombu is not to be confused with Celery. Kombu is a general purpose 
AMQP/messaging abstraction library. I don't think we agree on its potential 
role here. Perhaps it's better stated that I think Channel's minimalist API 
is too minimalist. I would prefer if additional AMQP-like abstractions 
existed such as topic routing and QoS.
 

>
> ASGI is essentially meant to be an implementation of the CSP/Go style of 
> message-passing interprocess communication, but cross-network rather than 
> merely cross-thread or cross-process as I believe that network transparency 
> makes for a much better deployment story and the ability to build a more 
> resilient infrastructure.
>

Again I don't agree with this argument and I don't see anything in Channels 
which backs up this claim. I believe this is where we likely have a 
fundamental disagreement. I see this network transparency as additional 
latency. I see the addition of the backend/broker as another moving part to 
break.

 
>
>> It’s hard for me to separate this work from the process by which it was 
>> created. Russ touched on my previous experience with the DEP process and I 
>> will admit that has jaded many of my interactions with the core team. 
>> Building consensus is hard and I’m posting this to help work towards the 
>> goal of community consensus. Thanks for taking the time to read this all 
>> the way through and I welcome any feedback.
>>
>
> I will put my hand up and say that this sidestepped the DEP process, and 
> that's entirely my fault. It was not my intention; I've been working on 
> this for over two years, and only last year did I go public with my 
> semi-final design and start asking for feedback; I should probably have 
> taken it into a DEP the

Re: My Take on Django Channels

[Mark Lavin]

Andrew,

I worked very hard to edit the tone of this message and I'm sorry if you 
felt anything in here was a personal attack. That certainly was not my 
intent. My natural speaking tendency leans toward hyperbole and I think 
there may have been places which got away from me here.

Best,

Mark

On Thursday, May 5, 2016 at 4:20:06 PM UTC-4, Andrew Godwin wrote:
>
>
>
> On Thu, May 5, 2016 at 12:34 PM, Mark Lavin  > wrote:
>
>> After somewhat hijacking another thread 
>> https://groups.google.com/d/msg/django-developers/t_zuh9ucSP4/eJ4TlEDMCAAJ 
>> I thought it was best to start fresh and clearly spell out my feelings 
>> about the Channels proposal. To start, this discussion of “Django needs a 
>> websocket story” reminds me very much of the discussions about NoSQL 
>> support. There were proof of concepts made and the sky is falling arguments 
>> about how Django would fail without MongoDB support. But in the end the 
>> community concluded that `pip install pymongo` was the correct way to 
>> integrate MongoDB into a Django project. In that same way, it has been 
>> possible for quite some time to incorporate websockets into a Django 
>> project by running a separate server dedicated for handling those 
>> connections in a framework such as Twisted, Tornado, Aiohttp, etc and 
>> establishing a clear means by which the two servers communicate with one 
>> another as needed by the application. Now this is quite vague and ad-hoc 
>> but it does work. To me this is the measuring stick by which to judge 
>> Channels. In what ways is it better or worse than running a separate server 
>> process for long-lived vs short-lived HTTP connections?
>>
>
> The main gains are (in my opinion):
>  - The same server process can serve both HTTP and WebSockets without path 
> prefixing (auto-negotiation based on the Upgrade header); without this you 
> need an extra web layer in front to route requests to the right backend 
> server
>  - HTTP long-polling is supported via the same mechanism (like WebSockets, 
> it does not fit inside the WSGI paradigm in a performant way)
>  - You get to run less processes overall
>
> That said, I don't see everyone running over to use Daphne in production, 
> which is why it's entirely reasonable to run two servers; one for HTTP and 
> one for WebSockets. Channels fully supports this, whether you run the HTTP 
> servers as self-contained WSGI servers or make them forward onto the ASGI 
> layer via the adapter.
>  
>
>>
>> At the application development level, Channels has the advantage of a 
>> clearly defined interprocess communication which would otherwise need to be 
>> written. However, The Channel API is built more around a simple queue/list 
>> rather than a full messaging layer. The choices of backends are currently 
>> limited to in-memory (not suitable for production), the ORM DB (not 
>> suitable for production), and Redis. While Redis PUB/SUB is nice for 
>> fanout/broadcast messaging, it isn’t a proper message queue. It also 
>> doesn’t support TLS out of the box. For groups/broadcast the Redis Channel 
>> backend also doesn’t use PUB/SUB but instead emulates that feature. It 
>> likely can’t use PUB/SUB due to the choice of sharding. This seemingly 
>> ignores robust existing solutions like Kombu, which is designed around AMQP 
>> concepts. Kombu supports far more transports than the Channel backends 
>> while emulating the same features, such as groups/fanout, and more such as 
>> topic exchanges, QoS, message acknowledgement, compression, and additional 
>> serialization formats.
>>
>
> Firstly, nothing in channels uses pub/sub - channels deliver to a single 
> reader of a queue, and thus cannot be built on a broadcast solution like 
> pub/sub.
>
> asgi_redis, the backend you're discussing, instead uses Redis lists 
> containing the names of expiring Redis string keys with data encoded using 
> msgpack, using LPOP or BLPOP to wait on the queue and get messages. It has 
> built-in sharding support based on consistent hashing (and with separate 
> handling for messages to and from workers).
>
> AMQP (or similar "full message queues") doesn't work with Channels for two 
> main reasons:
>
>  a) Running protocols through a queue like this requires incredibly low 
> latency; the Redis solution is on the order of milliseconds, which is a 
> speed I have personally not seen an AMQP queue reach
>
>  b) The return channels for messages require delivery to a specific 
> process, which is very difficult routing story given the AMQP design 
> structure. There's some solutions, but at the end of the day 

My Take on Django Channels

[Mark Lavin]

 deploying new code so you won’t drop HTTP 
connections. But the interface application definition and the worker code 
live in the same code base. It will be difficult to determine whether or 
not you need to restart the interface or not on a given deployment so many 
people will likely error on the side of restarting the interface as well. 
With a separate async server, likely in a separate code base, it would be 
easy to deploy them independently and only restart the websocket 
connections when needed. Also, it’s better if your application can handle 
gracefully disconnections/reconnections for the websocket case anyway since 
you’ll have to deal with that reality on mobile data connections and 
terrible wifi.

There is an idea floating around of using Channels for background 
jobs/Celery replacement. It is not/should not be. The message delivery is 
not guaranteed and there is no retry support. This is explicitly outside of 
the stated design goals of the project. Allowing this idea to continue in 
any form does a disservice to the Django community who may use Channels in 
this way. It’s also a slap in the face to the Celery authors who’ve worked 
for years to build a robust system which is superior to this naive 
implementation.

So Channels is at best on par with the existing available approaches and at 
worst adds a bunch of latency, potentially dropped messages, and new points 
of failure while taking up more resources and locks everyone into using 
Redis. It does provide a clear message framework but in my opinion it’s too 
naive to be useful. Given the complexity in the space I don’t trust 
anything built from the ground up without having a meaningful production 
deployment to prove it out. It has taken Kombu many years to mature and I 
don’t think it can be rewritten easily.

I see literally no advantage to pushing all HTTP requests and responses 
through Redis. What this does enable is that you can continue to write 
synchronous code. To me that’s based around some idea that async code is 
too hard for the average Django dev to write or understand. Or that nothing 
can be done to make parts of Django play nicer with existing async 
frameworks which I also don’t believe is true. Python 3.4 makes writing 
async Python pretty elegant and async/await in 3.5 makes that even better.

Sorry this is so long. Those who saw the DjangoCon author’s panel know that 
quickly writing walls of unreadable text is my forte. It’s been building 
for a long time. I have an unsent draft to Andrew from when he wrote his 
first blog post about this idea. I deeply regret not sending it and 
beginning to engage in this discussion earlier. It’s hard for me to 
separate this work from the process by which it was created. Russ touched 
on my previous experience with the DEP process and I will admit that has 
jaded many of my interactions with the core team. Building consensus is 
hard and I’m posting this to help work towards the goal of community 
consensus. Thanks for taking the time to read this all the way through and 
I welcome any feedback.

Best,

Mark Lavin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8c4b240d-2248-4af7-a8f6-f8ddff2d807e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Integration

[Mark Lavin]

Thank you Russ. I'll reconsider expressing my full thoughts on Channels 
more likely in another thread. For now I do think it's worth addressing 
this issue of benchmarks/performance which keeps being brought up. The 
argument is that since this is optional we don't need to see the benchmarks 
because there won't be any regressions, which is true. However, if it is 
also being said that this is so fundamentally important to Django and 
everyone will use it so it cannot live as an external project and must land 
in 1.10 then I don't think that argument can be made without ensuring there 
are no huge regressions moving an existing application from WSGI to ASGI. 
If nothing else those benchmarks seem important for Django users to make an 
informed choice about WSGI vs ASGI for their deployment. How can we not 
care about how this "fundamental change to Django" might impact performance 
or say that isn't a requirement to even measure, regardless of outcome, 
before its inclusion?

- Mark

On Wednesday, May 4, 2016 at 10:00:15 PM UTC-4, Russell Keith-Magee wrote:
>
> Hi Mark,
>
> On Thu, May 5, 2016 at 8:41 AM, Mark Lavin  > wrote:
>
>> Major features have never been perfect, no, but they have in the past 
>> typically gone through two paths to prove out their design/API/usefulness. 
>> One is as an established and mature third-party app such as messages, 
>> staticfiles, and django-secure. More recently the other has been through 
>> the DEP process: multiple templates (Jinja) and query expressions. Channels 
>> has done neither.
>>
>> Sorry if it seems that I've raised these issues late but I don't feel 
>> like there has been a good place for this discussion since the DEP process 
>> was circumvented. Most of the development for this has been in Andrew's 
>> space. I don't feel welcome to raise a dissenting opinion as a mere lowly 
>> member of the Django community. 
>>
>
> If that’s your perception, then we as a community clearly have a problem 
> that needs to be addressed. 
>
> You’ve been around the Django community since (AFAICT) 2009. You’re a 
> technical director at a well known and well respected Django consultancy. 
> You’ve given talks at DjangoCons. You’ve co-written a book about Django for 
> O’Reilly. If you’re not someone who is in a position to give an informed 
> opinion on issues with Channels, then I don’t know who is. If you feel like 
> you’re on the outer and your opinion is not welcome, then that’s *our* 
> failure, not yours.
>
> I can’t argue with the fact that the DEP process has been circumvented 
> here. I also acknowledge that this would be doubly frustrating given your 
> difficulties shepherding the content negotiation DEP. I don’t think I can 
> give a good answer for why this has been done, other than enthusiasm and 
> momentum overriding a not-entirely-well-established process.
>
> This thread (and email in general) probably isn’t the best place to flesh 
> out the solution to these process issues, but they definitely need to be 
> resolved. Discussion at PyCon and DjangoCon US is definitely called for - 
> I’ll be at both, and I’d definitely be interested in a BOF or similar 
> session to field opinions and thoughts from the community about this 
> process.
>  
>
>> It's pointless for me to continue to elaborate on the things I don't like 
>> about channels as I'm clearly in the minority and it's going to land. All I 
>> can do now is beg for these requirements to be optional.
>>
>
> Talking about channels specifically: This shouldn’t be true at all. 
> There’s certainly some momentum, but the code isn’t in trunk, so it’s not 
> too late. Even if it *was* in trunk - if you could demonstrate that there 
> was a serious problem, I’d support removing it from the codebase.
>
> I will admin that I haven’t been paying *close* attention to Andrew’s work 
> - I’m aware of the broad strokes, and I’ve skimmed some of the design 
> discussions, but I haven’t been keeping close tabs on things. From that 
> (admittedly weak) position, the only counterargument that I’ve heard are 
> performance concerns. 
>
> However, the last thing I heard on this subject was that the “raw WSGI” 
> path was essentially unmodified, so there shouldn’t be any particular 
> performance concerns from adding this to the codebase - just new 
> functionality for targeting websockets. For me, websockets are a big area 
> where Django is currently lacking, and everything I’ve read about Andrew’s 
> proposal has struck me as a reasonable compromise between Django’s need to 
> respond to a technical requirement, while maintaining ease of 
> implementation. 
>
> I take it that 

Re: Django Integration

[Mark Lavin]

Major features have never been perfect, no, but they have in the past 
typically gone through two paths to prove out their design/API/usefulness. 
One is as an established and mature third-party app such as messages, 
staticfiles, and django-secure. More recently the other has been through 
the DEP process: multiple templates (Jinja) and query expressions. Channels 
has done neither.

Sorry if it seems that I've raised these issues late but I don't feel like 
there has been a good place for this discussion since the DEP process was 
circumvented. Most of the development for this has been in Andrew's space. 
I don't feel welcome to raise a dissenting opinion as a mere lowly member 
of the Django community. It's pointless for me to continue to elaborate on 
the things I don't like about channels as I'm clearly in the minority and 
it's going to land. All I can do now is beg for these requirements to be 
optional.

- Mark

On Wednesday, May 4, 2016 at 6:48:03 PM UTC-4, Jacob Kaplan-Moss wrote:
>
> On Wed, May 4, 2016 at 2:45 PM, Marc Tamlyn  > wrote:
>
>> Major features merged into Django have generally never been as "perfect" 
>> as the standards required for smaller patches. There's a recognisation of 
>> the need for ongoing work, probably over the course of multiple versions, 
>> in order to perfect any major new feature. The effort involved in getting a 
>> patch like this to the point it can be merged at all is very significant. 
>> Many major patches (composite fields, templates widgets...) have previous 
>> not landed, sometimes multiple times, because they are nowhere near the 
>> standard that this channels patch is.
>>
>> My feeling is that the core team agree with Andrew that this is an 
>> important direction for Django, and 1.10 is the right release to include it 
>> in. Bug fixes, additional tests and so on can all be added between alpha 
>> and final as needed. The important thing is that we have broad agreement 
>> that the feature is good, the design is right, and an understanding that 
>> shortcomings will be worked on over the next year or two.
>>
>>
> I agree with this completely. I've been using Channels semi-seriously for 
> 2-3 months now, and in my opinion it easily clears the bar for inclusion. 
> The design is right, it's stable in production and under load, and has no 
> show-stopping bugs I've been able to find. Quite bluntly, it's probably 
> somewhat better than our average alpha-quality new feature. 
>
> My only real concern was around performance regressions, but now that the 
> WSGI "mode" hits no new code paths, that's no longer a concern. 
>
> Jacob 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/218e4513-91b1-4fad-b18c-b6b0e264fdbc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Integration

[Mark Lavin]

I can (and do) appreciate the effort that's gone into this work while still 
feeling as though it isn't ready to be merged with Django. To be honest 
given that this PR contains almost no changes to Django itself and only 
adds new code, I don't understand why it can't live outside of Django while 
it continues to mature.

Best,

Mark

On Wednesday, May 4, 2016 at 3:18:03 PM UTC-4, Andrew Godwin wrote:
>
>
>
> On Wed, May 4, 2016 at 11:52 AM, Mark Lavin  > wrote:
>
>>
>>
>> Given that there is no guarantee of message delivery, I don't think this 
>> is suitable or recommended for using for background tasks. I don't think 
>> the Django docs should encourage that usage. At least without a strong 
>> warning that this isn't a good idea.
>>  
>>
>>>
>>>> There are tests for the routing and request handling but there are no 
>>>> unittests that I can see for the auth/session handling,
>>>>
>>>
>>> Yes, these are missing.
>>>  
>>>
>>>> Channel/Message/Group/ChannelLayerManager/custom JSON encoder classes 
>>>> and their usage,
>>>>
>>>
>>> The ASGI conformance test suite takes care of this: 
>>> https://github.com/andrewgodwin/django/blob/channels/tests/channels_tests/test_database_layer.py#L3
>>>  
>>> (that's also used for the inmemory module and asgi_redis in their test 
>>> suites)
>>>
>>  
>>>
>>
>> Yes the code is covered by integration/functional tests but again I feel 
>> like there should be unittests for this as well.
>>
>
> Those are unit tests as far as I'm concerned, and they're quite 
> comprehensive - there's no smaller units of work to really write tests 
> _for_ in any of these backends.
>  
>
>>  
>>
>>> or the worker/runworker code.
>>>>
>>>
>>> What would you suggest these tests look like? They can't spin up a 
>>> worker and fire requests at it, and most of the worker code is interactive.
>>>
>>
>> It is certainly possible to fire up a worker in a subprocess and test it 
>> that way. Or you could create an instance of the Worker object and call its 
>> methods.
>>
>
> I'll look into this, but firing up subprocesses won't work as it suddenly 
> introduces Redis as a test dependency (right now all tests use the 
> in-memory backend).
>  
>
>>  
>>
>>>  
>>>
>>>> The changes to runserver aren't tested.
>>>>
>>>
>>> Runserver in Django only has a single test, presumably for the same 
>>> reason; I don't know how I can write a test for this considering the 
>>> autoreload and extra process mechanisms we'd want to cover.
>>>
>>  
>>>
>>>> The new test utilities aren't tested either (yes that's a thing).
>>>>
>>>
>>> I would say that the usage of those utilities in other tests means they 
>>> are tested sufficiently, given that they're used sufficiently; writing a 
>>> test for just the channel test case would look very similar to the basic 
>>> tests elsewhere.
>>>
>>
>> The existing test utilities (test client, asserting the number of 
>> queries, overriding settings, etc) have tests of their own 
>> https://github.com/django/django/blob/master/tests/test_utils/tests.py 
>> as do the extensions added by contrib.staticfiles. I don't see why Channels 
>> is special and therefore doesn't need them.
>>
>
> I would counter that by saying that the Channels test handler code is only 
> about 20 additional lines of logic and no new assertions, so the existing 
> suite is more than enough.
>  
>
>>  
>>
>>>  
>>>
>>>> All new code should have tests. Isn't that Django's current standard?
>>>>
>>>
>>> There's no clear standard, the docs just say the tests should "exercise 
>>> all of the new features". I admit that auth/session needs tests - I'll get 
>>> those done this week - but I don't see how we can reasonably test the 
>>> interactive commands in an automated fashion given the current test harness 
>>> (and lack of tests for things like runserver) in Django.
>>>
>>
>> To me it's a bad code smell when code is hard to test. I can't believe 
>> I'm having to convince anyone in this community about the value of testing.
>>
>
> Well, it is har

Re: Django Integration

[Mark Lavin]

I like Markus' suggestion and I think that's in line with how Django 
handles other optional dependencies such as the db bindings 
(psycopg2, MySQLdb, etc). Those raise an ImproperlyConfigured exception 
when there is an import error. The memcache cache backend on the other hand 
raises an import error if python-memcached isn't installed.

On Wednesday, May 4, 2016 at 1:09:40 PM UTC-4, Andrew Godwin wrote:
>
>
>
> On Wed, May 4, 2016 at 10:06 AM, Markus Holtermann <
> in...@markusholtermann.eu > wrote:
>
>> What about having asgiref and daphne as optional dependencies instead of
>> hard once and raising a proper exception "please install ..." when the
>> import fails?
>>
>> ```
>> try:
>>from asgiref import ...
>> except ImportError:
>>raise ImportError(
>>"Please ensure you installed asgiref to use this feature. Do so "
>> "with `pip install Django[channels]"
>>)
>> ```
>>
>> i.e. `pip install "Django[channels]"` ?
>>
>
> That is the other option, but it seems to run contrary to Django's 
> traditional batteries-included philosophy (and would result in some ugly, 
> ugly import code in the channels modules). I'm curious to hear more 
> opinions though, if there's support for it I'm happy to change it.
>
> Andrew 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c696555b-4d08-40e6-b1b5-f3e492b60249%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Integration

[Mark Lavin]

On Wednesday, May 4, 2016 at 12:39:02 PM UTC-4, Andrew Godwin wrote:
>
>
>
> On Wed, May 4, 2016 at 8:26 AM, Mark Lavin  > wrote:
>
>> As noted in the PR there is at least one new setting, 
>> CHANNEL_SESSION_ENGINE, which is lacking documentation.
>>
>
> That's been added now.
>  
>
>> The notes on deployment for "Running ASGI under WSGI" don't give any 
>> motivation why you would want to do this given that it doesn't support 
>> websockets in this case. Is there any advantages? Disadvantages?
>>
>
> It lets you still have background tasks and non-WebSocket interface 
> servers (or run websocket interfaces separately). I've expanded the docs to 
> say this.
>
 
>

Given that there is no guarantee of message delivery, I don't think this is 
suitable or recommended for using for background tasks. I don't think the 
Django docs should encourage that usage. At least without a strong warning 
that this isn't a good idea.
 

>
>> There are tests for the routing and request handling but there are no 
>> unittests that I can see for the auth/session handling,
>>
>
> Yes, these are missing.
>  
>
>> Channel/Message/Group/ChannelLayerManager/custom JSON encoder classes and 
>> their usage,
>>
>
> The ASGI conformance test suite takes care of this: 
> https://github.com/andrewgodwin/django/blob/channels/tests/channels_tests/test_database_layer.py#L3
>  
> (that's also used for the inmemory module and asgi_redis in their test 
> suites)
>
 
>

Yes the code is covered by integration/functional tests but again I feel 
like there should be unittests for this as well.
 

> or the worker/runworker code.
>>
>
> What would you suggest these tests look like? They can't spin up a worker 
> and fire requests at it, and most of the worker code is interactive.
>

It is certainly possible to fire up a worker in a subprocess and test it 
that way. Or you could create an instance of the Worker object and call its 
methods.
 

>  
>
>> The changes to runserver aren't tested.
>>
>
> Runserver in Django only has a single test, presumably for the same 
> reason; I don't know how I can write a test for this considering the 
> autoreload and extra process mechanisms we'd want to cover.
>
 
>
>> The new test utilities aren't tested either (yes that's a thing).
>>
>
> I would say that the usage of those utilities in other tests means they 
> are tested sufficiently, given that they're used sufficiently; writing a 
> test for just the channel test case would look very similar to the basic 
> tests elsewhere.
>

The existing test utilities (test client, asserting the number of queries, 
overriding settings, etc) have tests of their own 
https://github.com/django/django/blob/master/tests/test_utils/tests.py as 
do the extensions added by contrib.staticfiles. I don't see why Channels is 
special and therefore doesn't need them.
 

>  
>
>> All new code should have tests. Isn't that Django's current standard?
>>
>
> There's no clear standard, the docs just say the tests should "exercise 
> all of the new features". I admit that auth/session needs tests - I'll get 
> those done this week - but I don't see how we can reasonably test the 
> interactive commands in an automated fashion given the current test harness 
> (and lack of tests for things like runserver) in Django.
>

To me it's a bad code smell when code is hard to test. I can't believe I'm 
having to convince anyone in this community about the value of testing.
 

>  
>
>>
>> asgiref has one "basic" test around WSGI handling which the usage is 
>> documented in this change. I don't think anyone would dare say that covers 
>> potential edge cases in WSGI/HTTP handling. Not test related, the status 
>> code map is also missing many HTTP status codes.
>>
>
> The WSGI handling part of asgiref is not considered complete, and I will 
> probably remove the section about using it in the main Django docs. Only 
> the inmemory backend should be considered used by Django.
>
 
>

This is exactly my point. This project is not complete. That is not at all 
clear from this PR or from looking at the asgiref README which only 
re-enforces my feeling that this dependency should not be a requirement 
when installing Django.
 

>
>> I'm not in favor of on leeway for external dependencies for the reason 
>> you note: they are in the install_requires. Django has avoided having 
>> required dependencies and I don't think it sets a good precedent to have 
>> the f

Re: Django Integration

[Mark Lavin]

As noted in the PR there is at least one new setting, 
CHANNEL_SESSION_ENGINE, which is lacking documentation. The notes on 
deployment for "Running ASGI under WSGI" don't give any motivation why you 
would want to do this given that it doesn't support websockets in this 
case. Is there any advantages? Disadvantages?

There are tests for the routing and request handling but there are no 
unittests that I can see for the auth/session handling, 
Channel/Message/Group/ChannelLayerManager/custom JSON encoder classes and 
their usage, or the worker/runworker code. The changes to runserver aren't 
tested. The new test utilities aren't tested either (yes that's a thing). 
All new code should have tests. Isn't that Django's current standard?

asgiref has one "basic" test around WSGI handling which the usage is 
documented in this change. I don't think anyone would dare say that covers 
potential edge cases in WSGI/HTTP handling. Not test related, the status 
code map is also missing many HTTP status codes.

I'm not in favor of on leeway for external dependencies for the reason you 
note: they are in the install_requires. Django has avoided having required 
dependencies and I don't think it sets a good precedent to have the first 
set be those which don't meet the quality standards expected by the 
community. That is they don't have sufficient docs/tests to be include in 
Django itself. It isn't clear why these are required if you are claiming 
that they aren't required to run. If these we're in the install_requires 
then I would withdraw the objections about them with regard to this change 
in Django.

-Mark

On Wednesday, May 4, 2016 at 10:26:22 AM UTC-4, Andrew Godwin wrote:

>
> On Wed, May 4, 2016 at 6:15 AM, Mark Lavin  > wrote:
>
>> I had assumed this was still a work in progress because there are missing 
>> tests and some documentation. The build is passing but the unittest 
>> coverage for the new modules seems low or at least not up to the standards 
>> I expect for Django contributions. The same for the daphne and asgiref 
>> packages which are new requirements. In previous discussions there were 
>> talks about performance benchmarks to track potential regressions before 
>> this would be accepted similar to the template-based widget rendering 
>> change. I don't see any references here or in this work. What is the status 
>> of those?
>>
>>
> - The documentation on Channels is considerably more than I would have 
> accepted for a patch. What do you think is missing?
>
> - The channels branch is likely missing some tests around auth and session 
> in particular, but a lot of the "main" stuff has tests; what would you need 
> before accepting it?
>
> - asgiref quite literally has more tests than actual code we're using, so 
> I don't see a problem there
>
> - Daphne is severely lacking in tests for WebSocket encoding/decoding 
> support as writing a test harness for it is ridiculously hard and I was 
> hoping to get some help on that now we finally have the MOSS money around.
>
> - There are no performance regression tests as by default the codepath in 
> Django is the same as before (apart from runserver, which in my basic tests 
> actually sped up); as long as you deploy using WSGI, you never even touch 
> Channels code (which wasn't the original plan, but now is for sanity 
> reasons).
>
> I would like some leeway on the external dependencies being fully tested 
> considering that they are not beholden to Django's release cycle, but maybe 
> because we're declaring them as core dependencies (I patched in 
> install_requires into setup.py for them) we should be very strict? If this 
> patch is going to be denied based on Daphne not having sufficient test 
> coverage then I suspect we'll miss the deadline and have to foist this on 
> the LTS instead, which I really don't want.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0b519047-96ee-4ac9-9230-0f487625a7e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Integration

[Mark Lavin]

I had assumed this was still a work in progress because there are missing 
tests and some documentation. The build is passing but the unittest 
coverage for the new modules seems low or at least not up to the standards 
I expect for Django contributions. The same for the daphne and asgiref 
packages which are new requirements. In previous discussions there were 
talks about performance benchmarks to track potential regressions before 
this would be accepted similar to the template-based widget rendering 
change. I don't see any references here or in this work. What is the status 
of those?

Best,

Mark

On Tuesday, May 3, 2016 at 9:04:19 PM UTC-4, Andrew Godwin wrote:
>
> I'm basically slowly fixing the test system failures (mostly because I 
> introduced some new packages); the patch is ready to go code-wise, just 
> need to get it green. Let me fix the current set now and see if that pushes 
> it over the edge.
>
> (There are some refinements the patch needs in terms of things like making 
> the enforce_ordering decorator a bit more efficient, but I see these as 
> being entirely valid things to do after the feature freeze)
>
> Andrew
>
> On Tue, May 3, 2016 at 5:57 PM, Tim Graham  > wrote:
>
>> Hi Andrew,
>>
>> How are things going with the patch [0]? Do you think you might have it 
>> ready by next Monday or so, so that I'll have at least a few days to review 
>> it before the alpha scheduled for May 16?
>>
>> [0] https://github.com/django/django/pull/6419
>>
>> On Saturday, March 5, 2016 at 7:49:41 PM UTC-5, Andrew Godwin wrote:
>>>
>>> My intention is still to get it in - the external library is pretty much 
>>> at the stable point now, and I'm preparing to brand it 1.0. Once I do that, 
>>> I'll start work on the Django patch.
>>>
>>> It's made a bit easier by the fact that the code is four repositories, 
>>> three of which (asgiref, daphne, and asgi_redis) can continue to be 
>>> separate modules. Only the code in the "channels" repository needs to go 
>>> into Django itself, and while it's the biggest repo, most of it should slot 
>>> right in; there's not really much in core itself that needs changing, more 
>>> about addition and tweaking some imports.
>>>
>>> Andrew
>>>
>>> On Sat, Mar 5, 2016 at 4:44 PM, Tim Graham  wrote:
>>>
 Hi Andrew,

 What's your thinking about whether or not any of this will make it into 
 1.10? The alpha is scheduled for May 16.

 On Saturday, March 5, 2016 at 2:40:05 PM UTC-5, Andrew Godwin wrote:
>
> Hi Chad,
>
> The REST framework request feature integration, in particular, has not 
> started yet; Mozilla has been having some issues working out how to pay 
> their various grantees, and from my understanding we still haven't 
> received 
> the grant money yet.
>
> Channels is well underway, but that's only because I'm able to do the 
> work now and wait to get paid for it later; I don't expect the same of 
> other contributors.
>
> Andrew
>
> On Fri, Mar 4, 2016 at 3:58 PM, Chad Paulson  
> wrote:
>
>> I was happy to read that part of the Mozilla Open Source Support 
>> program funding that was recently awarded to the Django Software 
>> Foundation 
>> will go toward integrating key components of Django REST Framework into 
>> Django.
>>
>> Since I haven't found any update since the initial announcement in 
>> December, I was curious what the status of this integration is and, if 
>> possible, how soon it might be available.
>>
>>
>> https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/
>>
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, 
>> send an email to django-develop...@googlegroups.com.
>> To post to this group, send email to django-d...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/2bdf074c-5e21-48cf-a93c-ba19f7744a89%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> -- 
 You received this message because you are subscribed to the Google 
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at https://groups.google.com/group/django-developers.
 To view this discussion on the

Re: Channels integration plan, first draft

[Mark Lavin]

> You seem to be assuming I'm here to foist a brand new middle layer on
everyone; I'm not. I'm here to make one that fits neatly into Django, that
I think most people will want to turn on, and that provides a lot of value
in exchange for a slight round-trip performance hit - my goal is sub-5ms,
and preferably sub-3. If it starts being 10/20/30 milliseconds of cost,
then we'll have to change our approach until it's acceptable.

Yes that's how I read this plan and that's why I think it needs some
clarity. I didn't mean for this to turn into a long discussion about
performance. This was meant to be a discussion about the transition plan.
To go back to my original message, I see no gain for existing WSGI
applications to have this on by default, even using the in-memory channel, when
they upgrade to 1.10 (or whenever this lands). The current plan reads as
though it will. From what you are saying above this sounds more like a
django.contrib.channels than a django.core.channels. Either way I feel the
plan should provide more clarity in that regard.

Andrew, I thank you for your patience and civility in these discussions. I know
this is something you've been working hard on and I'm not trying to be
needlessly critical of your work.

On Fri, Dec 18, 2015 at 10:07 AM, Andrew Godwin  wrote:

>
>
> On Fri, Dec 18, 2015 at 2:00 PM, Mark Lavin  wrote:
>
>> Anssi criticisms are fair and I feel that some of these responses are
>> glossing over the details.
>>
>
> I'm sorry if it comes across like that - a lot of these are things I've
> been considering for a while and so I can forget to provide context.
>
>
>> You've claimed this is the same or equivalent to a forked worker model
>> but it isn't because there is no process management/link between the
>> interface and worker and because you've chosen to make this network
>> transparent. As much as you'd like to claim this isn't like Celery, the
>> same issues that exist when trying to (ab)use Celery for blocking RPC calls
>> is what you will have here.
>>
>
> I never claimed this isn't like Celery; it is quite a bit like it, but
> with specific changes (no guaranteed delivery, no single-response
> mechanism) that make it better at throughput.
>
>
>>
>> > Then the response is dropped, the same way a WSGI worker drops a
>> connection if it dies.
>>
>> It isn't entirely the same because in this channel case the worker
>> doesn't know the client/interface dropped the connection. It's still
>> working hard to generate a response which will sit in the response channel
>> (Redis, memory, etc) until it expires (assuming that all backends expire
>> channel messages). That doesn't happen in the current WSGI interface.
>>
>
> True, though from what I understand of the WSGI spec you also don't know
> the client has disconnected until you try to write out content to it. Most
> views would still run the entire thing and make a response, only to drop it.
>
> There is also a http.disconnect message type planned to be implemented for
> when a client disconnects before the response is entirely read, but that's
> more for long-poll usage where you want to keep track of who has an open
> connection.
>
>
>>
>> > All Django requests already involve multiple round trips to database
>> servers, and the Redis backend at least is much quicker on the processing
>> side.
>>
>> This isn't true. Database round trips are not a requirement for Django's
>> current architecture. I'll concede that many views touch the DB but that's
>> a choice of the developer, not Django's. When using channels the views will
>> still need to make the same DB calls, though that processing happens at the
>> worker. Putting Redis in between doesn't make it faster.
>>
>
> Right, and channels will not be a requirement for Django's future
> architecture. They'll just be a thing that I expect most people to turn on
> as they provide a feature set a lot of types of sites need. And I never
> said it would be faster - you'll see me repeatedly say channels provides no
> performance gain - at best, it might help smooth response times with the
> way the workers load balance jobs based on when they're free.
>
>
>>
>> > You could make the same argument about not having separate load
>> balancers and nginx serving static files - after all, you're just adding
>> another network roundtrip to send traffic onward from the other server to
>> Django.
>>
>> There are notable differences here. These are persistent HTTP connections
>

Re: Channels integration plan, first draft

[Mark Lavin]

Anssi criticisms are fair and I feel that some of these responses are
glossing over the details. You've claimed this is the same or equivalent to
a forked worker model but it isn't because there is no process
management/link between the interface and worker and because you've chosen
to make this network transparent. As much as you'd like to claim this isn't
like Celery, the same issues that exist when trying to (ab)use Celery for
blocking RPC calls is what you will have here.

> Then the response is dropped, the same way a WSGI worker drops a
connection if it dies.

It isn't entirely the same because in this channel case the worker doesn't
know the client/interface dropped the connection. It's still working hard
to generate a response which will sit in the response channel (Redis,
memory, etc) until it expires (assuming that all backends expire channel
messages). That doesn't happen in the current WSGI interface.

> All Django requests already involve multiple round trips to database
servers, and the Redis backend at least is much quicker on the processing
side.

This isn't true. Database round trips are not a requirement for Django's
current architecture. I'll concede that many views touch the DB but that's
a choice of the developer, not Django's. When using channels the views will
still need to make the same DB calls, though that processing happens at the
worker. Putting Redis in between doesn't make it faster.

> You could make the same argument about not having separate load balancers
and nginx serving static files - after all, you're just adding another
network roundtrip to send traffic onward from the other server to Django.

There are notable differences here. These are persistent HTTP connections
and don't suffer from the same problem of client/server drops previously
noted. They are also cacheable in a known way allowing round-trips or
bandwidth to be avoided. Many Django applications will continue to use load
balancers with channels. It can't be denied that channels introduce two
more network round trips that didn't exist before. Trying to paint this as
"we already talk over the network, so what's a couple more" is not a
compelling argument to me.

On Fri, Dec 18, 2015 at 5:01 AM, Andrew Godwin  wrote:

>
>
> On Fri, Dec 18, 2015 at 7:34 AM, Anssi Kääriäinen 
> wrote:
>
>> I have a gut feeling this isn't going to work that well. The reasons
>> include:
>>   - Backwards compatibility: how is a large site going to upgrade from
>> 1.9 to 1.10?
>>
>
> None of the core view API will change. A 1.9 codebase will boot and work
> on 1.10/channels with no code changes.
>
>
>>   - Complexity of setup.
>>
>
> Could you elaborate? If you don't want it, you don't need to configure
> anything, and if you do, for most people it's just getting a Redis server
> running and pointing a setting at it.
>
>
>>   - Error conditions: for example, what happens when an interface server
>> sends a request to worker, and then dies (that is, the response channel has
>> zero listeners). Similarly for chunked messages.
>>
>
> Then the response is dropped, the same way a WSGI worker drops a
> connection if it dies. Chunked responses will time out after a certain
> period and be dropped too, in the same way that a deadlocked normal server
> would.
>
>
>>   - Does the architecture really scale enough? The channel backend is
>> going to be a bottleneck, it needs the ability to handle a huge amount of
>> data and a huge amount of individual messages. In particular, the request
>> channel is going to be contested. We know classic http scales, but is the
>> same true for interface server architecture?
>>
>
> I believe it will - have you read through the sharding and scaling plan in
> the docs? Channels is carefully designed to have no state in anything but
> interface servers, all workers handling all message types and queuing of
> messages exactly so you can scale horizontally; you can divide a very large
> site into several clusters of interfaces and workers fronted by load
> balancers, and each cluster would have multiple Redis (e.g.) backends with
> requests equally sharded across them using consistent hashing.
>
>
>>   - Performance. Each request and response needs two additional network
>> roundtrips. One to save to the channel server, one to fetch from the
>> channel server. If the messages are large, this adds a lot of latency.
>>
>
> All Django requests already involve multiple round trips to database
> servers, and the Redis backend at least is much quicker on the processing
> side. You could make the same argument about not having separate load
> balancers and nginx serving static files - after all, you're just adding
> another network roundtrip to send traffic onward from the other server to
> Django.
>
>
>>   - Untested architecture: does any big site use this kind of
>> architecture for all http handling?
>>
>
> I agree with you here - see below for my justification. I've seen it used
> for other things (data update netw

Re: Channels integration plan, first draft

[Mark Lavin]

I have concerns about "built-in feature" means for existing applications 
(and future applications really). Under "Preserving Simplicty" you note 
that you should be able to run Django as simply as you do now. Is there a 
consideration to run "classic" WSGI applications without channels? 
This in-memory channel isn't free. It still requires serializing the 
incoming HTTP request to JSON, putting it into the channel queue, 
deserializing the JSON, , serializing the 
response to JSON, putting it into the response channel, deserializing the 
response, and sending to the client. That's two JSON serialization round 
trips which don't currently exist with no gain for existing applications. 
As you note later on this currently breaks non-root mounted (those using 
SCRIPT_NAME) applications but doesn't appear to have a plan to resolve it.

Best,

Mark


On Thursday, December 17, 2015 at 6:35:18 AM UTC-5, Andrew Godwin wrote:
>
> Hi everyone,
>
> One of the first steps I want to get done for Channels is get a rough plan 
> in place for how things are going to work in terms of where code goes, 
> supported versions, etc. I've written up my thoughts on this into a first 
> draft of what I'm calling the "integration plan":
>
> http://channels.readthedocs.org/en/latest/integration-plan.html
>
> Feedback is much welcomed - I'd particularly like to hear people's 
> thoughts on the ideas of releasing both natively in Django 1.10 and as a 
> third-party addon for 1.8/1.9, plus the new (for Django) concept of 
> releasing part of it as a separate codebase, though still under the Django 
> umbrella.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ab27553c-7ebe-4a97-83ad-0fcd4dcc5023%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Content Negotiation implement

[Mark Lavin]

Hi,

Tom has done a great job describing why this has stalled recently. I plan 
to pick back up on this now that the book has been released. I agree that 
defining the public API is the best first step and the current DEP is not 
sufficient in this regard. Before making any internal changes we need to 
define how the user will define new content types: where and at what level. 
We also need to define how the user will describe multiple response types 
for a view.

At a code level it's worth digging into how the current form-encoded and 
multi-part data is detected and parsed in Django. It's also worth reviewing 
how this problem has been solved by the community before: django-piston, 
django-tasypie, django-rest-framework, django-nap, restless, etc to look 
for common patterns and solutions as well as things that haven't worked 
well in hindsight.

Best,

Mark

On Monday, November 10, 2014 5:32:54 AM UTC-5, Tom Christie wrote:
>
> > I'll be trying to enhance my knowledge base about django request 
> response and other related stuffs necessary if you please show me some 
> light to get started or from where actually should I start.
>
> I'd generally think that a decent place to start would be to write up what 
> some of the documentation *before* actually implementing any code.
> Figure out where would you describe `request.data` in the Django docs, 
> figure out what other bits of the DEP would need to be public documented 
> API, ignoring the under-the-hood implementation details what does the user 
> need to know about them and how they work?
>
> An alternative way to get involved without necessarily diving right into 
> the DEP might be to review any outstanding tickets to do with HTTP handling 
> and requests and responses. Even if you don't feel able to resolve any of 
> those issues having a browse through and trying to understand them would be 
> a good way to start getting to grips with the Django codebase.
>
> Hope that helps,
>
>   Tom
>
> On Sunday, 9 November 2014 13:36:53 UTC, Asif Saifuddin wrote:
>>
>> Thank you both for your kind response.
>>
>> I have only almost about a year working with django and still learing. My 
>> django and python skills are still so limited as far I know. Reading the 
>> dep and other related links of discussions and browsing some source code 
>> things are bit overwhelming to me right now :p  but I am willing to expand 
>> my technical knowledge  by learning and trying to contribute on this part.
>>
>> I have some limitations in design decisions and understanding the overall 
>> implementation process But still I want to start working on this part 
>> primarily with you guys helps and suggestions.
>>
>> I'll be trying to enhance my knowledge base about django request response 
>> and other related stuffs necessary if you please show me some light to get 
>> started or from where actually should I start.
>>
>> Kind Regards
>>
>> Asif 
>>
>> On Sun, Nov 9, 2014 at 4:31 PM, Asif Saifuddin  wrote:
>>
>>> Thank you both for your kind response.
>>>
>>> I have only almost about a year working with django and still learing. 
>>> My django and python skills are still so limited as far I know. Reading the 
>>> dep and other related links of discussions and browsing some source code 
>>> things are bit overwhelming to me right now :p  but I am willing to expand 
>>> my technical knowledge  by learning and trying to contribute on this part.
>>>
>>> I have some limitations in design decisions and understanding the 
>>> overall implementation process But still I want to start working on this 
>>> part primarily with you guys helps and suggestions.
>>>
>>> I'll be trying to enhance my knowledge base about django request 
>>> response and other related stuffs necessary if you please show me some 
>>> light to get started or from where actually should I start.
>>>
>>> Kind Regards
>>>
>>> Asif 
>>>
>>>
>>> On Saturday, November 8, 2014 3:01:28 PM UTC+6, Asif Saifuddin wrote:

 Hi,
  when https://github.com/django/deps/blob/master/drafts/
 content-negotiation.rst this dep is to be implemented in django? 1.8?

 Regards

 Asif

>>>  -- 
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/django-developers/g1P7H1PhZaY/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to 
>>> django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/436d3c71-2567-462d-a035-9880511124ab%40googlegroups.com
>>>  
>>> 

Re: Migrations and Reusable Apps

[Mark Lavin]

Support of south_migrations directory in South goes a long way to 
addressing this problem from my perspective so that's very good news to me.

On Tuesday, June 17, 2014 1:34:02 PM UTC-4, Trey Hunner wrote:
>
> -BEGIN PGP SIGNED MESSAGE- 
> Hash: SHA1 
>
> On 06/17/2014 10:05 AM, Andrew Godwin wrote: 
> > 1.7 is not going to be a pretty release for third-party apps for a 
> > number of reasons: this, the app-loading refactor, the changes to custom 
> > fields, and a few other things. I might leave an open offer of help 
> > upgrading from myself after the release to any app maintainers to try 
> > and smooth that process along. 
>
> +1 that.  This has been the hardest recent release for my own apps 
> (mostly because of the app-loading refactor). 
>
> I'm grateful for the help I received while migrating 
> django-simple-history to 1.7. 
>
> Sign me up for that offer also.  I can help out with 1.7 support advice 
> as my abilities allow. 
>
> - -- 
> Trey Hunner 
> http://treyhunner.com 
> -BEGIN PGP SIGNATURE- 
> Version: GnuPG v1 
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ 
>
> iQEcBAEBAgAGBQJToHv2AAoJEOpnfp/NreonGzoIAMOpD4xBRiZDDfw1WwuHqVBd 
> mrq/VbyA7iTGMzQZ0ZSXhUT4Qd1dR0T1k0Qcyaeg837GdufkL4ikj8mFXHIRVAes 
> pXHYkV2W3NI8J6VqPDw1fB41nNcFxa7q/1GoABT0yLsAAfCu6v2WNpx+qipFmI7g 
> 6ugpL7fa+9lqd2K2bDsgW9s0UR/Z67DenpCdmYN9kFk8h49c+B/v/EbDV89qT3tg 
> pogtGKE/UOsvQG76v4BEcQmnQkAKNPzY9jFFb8wkhgzwkBj6tTXhbuqRw2TDhkKH 
> ORJj5xEKrIeSflFHqb4wT2zjlOMpyW3dwDs+X0PUe8mU0ZhPhpvtVFgWjA1aDOw= 
> =Marr 
> -END PGP SIGNATURE- 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4e88c232-aa3c-4cdc-a9ec-feb3a7493d06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Migrations and Reusable Apps

[Mark Lavin]

This might be a solution for projects which wish to upgrade in advance of 
their dependencies but as you note its implications for future app upgrades 
are unclear.

This still doesn't provide an answer for app developers. The deprecation 
timeline says migrations aren't required until 2.0. But if your app depends 
on contrib.auth it will be required in 1.7 based on this previous note.
As best I can tell, supporting Django < 1.7 + South and Django 1.7+ 
requires making a backwards incompatible release of the app. That's fine if 
that is the only way forward but I think the community could use more
guidance on this.

On Tuesday, June 17, 2014 10:06:11 AM UTC-4, Tim Graham wrote:
>
> I'm sure Andrew will have more info, but you could use MIGRATION_MODULES 
> and generate migrations for any 3rd party apps that don't have them. I'm 
> not sure how the transition when the app starts shipping its own migrations 
> will go.
>
> On Tuesday, June 17, 2014 9:18:57 AM UTC-4, Mark Lavin wrote:
>>
>> Hello,
>>
>> I noticed some changes over the past few days to the migrations and I was 
>> concerned about how this could impact reusable applications planning to 
>> support 1.7.
>> In particular there is a note in 
>> https://docs.djangoproject.com/en/1.7/topics/migrations/#dependencies
>>
>> Be aware, however, that unmigrated apps cannot depend on migrated apps, 
>> by the very nature of not having migrations. This means that it is not 
>> generally possible to have an unmigrated app have a ForeignKey or 
>> ManyToManyField to a migrated app; some cases may work, but it will 
>> eventually fail.
>>
>>
>> Given that contrib.auth now has migrations and many reusable applications 
>> have FKs to the User model, doesn't this force most reusable applications 
>> to ship migrations
>> in order to be compatible with 1.7? Is this an unintended consequence or 
>> an expected requirement? I know the plan is to eventually require 
>> migrations but this
>> seems like a hard requirement which could make upgrading to 1.7 more 
>> difficult due to lagging dependencies.
>>
>> Is there an acceptable approach for a reusable application to support 1.6 
>> + South and 1.7 + db.migrations? This 
>> http://treyhunner.com/2014/03/migrating-to-django-1-dot-7/ seemed
>> like a good approach though it does highlight the backwards compatibility 
>> issues.
>>
>> Best,
>>
>> Mark
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ec0ab6f4-5aa7-4784-9a17-e957696268f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Migrations and Reusable Apps

[Mark Lavin]

Hello,

I noticed some changes over the past few days to the migrations and I was 
concerned about how this could impact reusable applications planning to 
support 1.7.
In particular there is a note in 
https://docs.djangoproject.com/en/1.7/topics/migrations/#dependencies

Be aware, however, that unmigrated apps cannot depend on migrated apps, by 
the very nature of not having migrations. This means that it is not 
generally possible to have an unmigrated app have a ForeignKey or 
ManyToManyField to a migrated app; some cases may work, but it will 
eventually fail.


Given that contrib.auth now has migrations and many reusable applications 
have FKs to the User model, doesn't this force most reusable applications 
to ship migrations
in order to be compatible with 1.7? Is this an unintended consequence or an 
expected requirement? I know the plan is to eventually require migrations 
but this
seems like a hard requirement which could make upgrading to 1.7 more 
difficult due to lagging dependencies.

Is there an acceptable approach for a reusable application to support 1.6 + 
South and 1.7 + db.migrations? 
This http://treyhunner.com/2014/03/migrating-to-django-1-dot-7/ seemed
like a good approach though it does highlight the backwards compatibility 
issues.

Best,

Mark

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/afd7da9e-02e3-4c80-bef9-71dac036c415%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: DEPs: Django Enhancement Proposals

[Mark Lavin]

I'll be the first one to bite and submit 
one https://github.com/django/deps/pull/1 It needs a lot of work as noted 
in the PR but I'm happy to be a guinea pig for the process.

Best,

Mark

On Monday, April 14, 2014 8:41:07 PM UTC-4, Russell Keith-Magee wrote:
>
> On Tue, Apr 15, 2014 at 2:17 AM, Aymeric Augustin 
> > wrote: 
> > On 14 avr. 2014, at 19:51, Adrian Holovaty 
> > > 
> wrote: 
> > 
> >> One idea was to start doing a Django equivalent of Python's PEPs, which 
> would serve as a formal way to document large new features/changes in 
> Django. 
> > 
> > Would it be interesting to retro-spec the major features added in the 
> past? 
>
> I agree - there are a bunch of wiki pages that capture a lot of this 
> information; it wouldn't be a whole lot of effort to turn this into 
> formal DEP documents. 
>
> https://code.djangoproject.com/wiki/ClassBasedViews 
> https://code.djangoproject.com/wiki/ContribAuthImprovements 
> https://code.djangoproject.com/wiki/ContribEmailAuth (not yet finalised) 
> https://code.djangoproject.com/wiki/AppLoadingRefactor (partial; 
> doesn't include everything Aymeric did, but there are other mailing 
> list threads) 
> https://code.djangoproject.com/wiki/FileStorageRefactor 
> https://code.djangoproject.com/wiki/LoggingProposal (includes link to 
> mailing list with the full details) 
> https://code.djangoproject.com/wiki/ReplacingGetAbsoluteUrl (proposed, 
> but never actioned) 
> https://code.djangoproject.com/wiki/SchemaEvolution (older; probably 
> not relevant) 
>
> Russ %-) 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a6c0d818-3683-4bfd-8834-9379ff92c388%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Deprecating Widget Media class

[Mark Lavin]

In my experience using the Media class is really only helpful in one place: 
the admin. If you have a third-party widget which requires CSS or JS is a 
pain to include it without the use of Media. Users want to just include the 
widget in the form class and have it "just work". A majority of the 
issues/questions I deal with on django-selectable are about how to get it 
working in the admin. Telling people to override and include it in 
the admin/base_site.html only seems adds confusion. It is documented how to 
override the admin templates but it would seem no one actually reads or 
understands them. In some cases they are trying to use apps which are 
already overriding it (like django-admin-tools). Outside the admin the 
Media class isn't nearly as helpful. It's easy enough to document adding 
the CSS/JS references in your base.html and users seem to understand that.

The biggest shortcoming of the Media class is summarized 
in https://code.djangoproject.com/ticket/12265. The order in which it is 
added isn't helpful nor well documented. I wouldn't complain too loudly if 
it was removed. It would be nice if it was replaced with another hook 
(possibly only in the admin) to improve the story for a third-party widget 
which requires some CSS/JS.

Best,

Mark

On Tuesday, April 1, 2014 2:32:15 PM UTC-4, Tim Graham wrote:
>
> There's been some discussion in ticket 
> #22298about deprecating the 
> widget Media class. If this would make you 
> particularly happy or sad, please leave your feedback on this thread or on 
> the ticket.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/76e70be1-35fa-4a28-ae05-0529bce15685%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Migrations in Django 1.7 make unit testing models harder

[Mark Lavin]

I don't see any meaningful notes about apps being required to ship 
migrations beginning with 1.9. I do see deprecation notes about the syncdb 
signals changing and the syncdb command itself is clearly deprecated. This 
legacy behavior is handled by sync_apps in the migrate command but there 
aren't any deprecation warnings raised when it executes. I don't see 
anything which points to this behavior being deprecated other than this 
thread.

On Tuesday, March 25, 2014 4:12:42 PM UTC-4, Andrew Godwin wrote:
>
> So, the functionality whereby you can have apps which do not use 
> migrations (i.e. that use the old creation backends) is meant to go away in 
> 1.9 (i.e. the standard three-release deprecation cycle). Most of the 
> side-effects of this are detailed in 
> https://docs.djangoproject.com/en/dev/internals/deprecation/#deprecation-removed-in-1-9but
>  not this one - the fact that setting an empty MIGRATION_MODULE means it 
> falls back to the old syncdb method (in 1.9, this would just make Django 
> think that the app had no migrations at all and do nothing, and probably 
> raise a warning or error).
>
> The setting itself isn't deprecated, just this undocumented behaviour, so 
> I haven't put a warning next to it in the docs.
>
> As for the general TDD problem, we do have the squashing functionality now 
> so the --update problem is not quite as bad, but I'm not averse to 
> something like it appearing in 1.8 (1.7 is too frozen now to add it IMO).
>
> Andrew
>
>
> On Tue, Mar 25, 2014 at 12:01 PM, Shai Berger 
> > wrote:
>
>> On Tuesday 25 March 2014 20:36:39 Marc Tamlyn wrote:
>> > Do we have an equivalent of south's --update? This would mean you don't 
>> get
>> > many files. We don't want to make it too hard for people to work in a
>> > strict TDD fashion.
>> >
>> +1
>>
>> Shai.
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to 
>> django-d...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/201403252101.52683.shai%40platonix.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/262a32fd-fd5c-495b-a0ba-878372d3f7ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Migrations in Django 1.7 make unit testing models harder

[Mark Lavin]

Andrew,

Can you clarify exactly what is deprecated in this work-around? I don't see 
anything in the deprecation timeline to remove MIGRATION_MODULES nor any 
pending deprecations related to its usage. It seems like could probably be 
replaced by something that uses the app-loading/app-configs instead but it 
does look like it's been done.

Best,

Mark

On Tuesday, March 25, 2014 2:14:55 PM UTC-4, Andrew Godwin wrote:
>
> Yes, the new behaviour is by design, in the sense that the workaround you 
> mentioned will be deprecated in 1.9 (along with all syncdb-related 
> functionality). This way, tests always run against the version of your 
> models that production would run, so you don't run the risk of the tests 
> passing locally as they're not using migrations, pushing it live, and then 
> things failing in production as the tables don't match.
>
> I'm happy to take a small patch to the docs to mention that you need to 
> run makemigrations for tests to see your database changes, but the 
> workaround shouldn't be in there - as I said, it's already part of the 
> deprecation cycle.
>
> Andrew
>
>
> On Tue, Mar 25, 2014 at 10:21 AM, Bernie Sumption 
> 
> > wrote:
>
>> Hi Django devs,
>>
>> I've just started a new project in 1.7b, and the development experience 
>> working with unit tests on models is more complicated than it was in 1.6. 
>> It's all down to how the throwaway test databases are created. In 1.6, the 
>> create_test_db function "Creates a new test database and runs syncdb 
>> against it." In 1.7, it runs "migrate".
>>
>> While generally speaking, migrate is the new syncdb, this behaviour is 
>> not ideal for tests. In 1.6 "syncdb" created a database reflecting the 
>> current state of the models in models.py. "migrate" creates a database 
>> reflecting the state of the models at the last time makemigrations was run. 
>> If you're doing TDD and constantly making small changes to your models then 
>> runnning unit tests, you have to run makemigrations before each test run to 
>> get your tests to work. You therefore end up with many tiny migration files 
>> representing the minute-by-minute history of development.
>>
>> I came up with a pretty effective workaround that is working for me, but 
>> I thought I'd post this here as others are sure to encounter this issue, 
>> and I think that it makes more sense for the behaviour produced by this 
>> workaround to be the default for running tests.
>>
>> If makemigrations has not yet been run, the "migrate" command treats an 
>> app as unmigrated, and creates tables directly from the models just like 
>> syncdb did in 1.6. I defined a new settings module just for unit tests 
>> called "settings_test.py", which imports * from the main settings module 
>> and adds this line:
>>
>> MIGRATION_MODULES = {"myapp": "myapp.migrations_not_used_in_tests"}
>>
>> Then I run tests like this:
>>
>> DJANGO_SETTINGS_MODULE="myapp.settings_test" python manage.py test
>>
>> This fools migrate into thinking that the app is unmigrated, and so every 
>> time a test database is created it reflects the current structure of 
>> models.py.
>>
>> So my feature request is as follows:
>>
>> If the new behaviour is by design and considered desirable, then it is a 
>> big change from the previous version and should be prominently documented 
>> in the migrations and testing pages, along with the workaround. I'm happy 
>> to write this documentation if that's the way you want to go.
>>
>> However, if the new behaviour is not by design but just a by-product of 
>> the new migrations feature, I suggest making the workaround the default 
>> behaviour. I don't (yet!) know enough about Django internals to volunteer 
>> for this however.
>>
>> Thanks for your time,
>>
>> Bernie :o)
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to 
>> django-d...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/00e9a053-3e61-4c5d-8fcc-5a4d67deab38%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/

Re: #20824 - Email-based auth contrib module; some help required

[Mark Lavin]

Welcome to the pain that third party app maintainers have already been 
dealing with for the past year. I ran into this problem as well when I 
added custom user support to one of the apps I maintain. As you noted you 
can't simply change the AUTH_USER_MODEL setting because once the test DB is 
created the FK references are set. The only solution I found was to use 
tox (which I was already using to test versus multiple Python/Django 
versions) and run the full suite once with the standard user model and 
again with a custom user model. Some tests are skipped when the default 
user is used and others are skipped with the custom user is in use. You can 
see it in action here: https://github.com/mlavin/django-all-access

I'd love to see a more general solution to this but I have a feeling 
anything that attempts to modify the DB tables between tests is destined to 
fail.

Best,

Mark

On Wednesday, February 26, 2014 2:33:24 AM UTC-5, Russell Keith-Magee wrote:
>
> Hi all,
>
> tl;dr; I've been working on a last little feature for 1.7 - an email-based 
> User model for contrib, but I need some help/guidance on the last step - 
> getting the tests working.
>
> The idea here is that we just ship a normal auth.User, but with email 
> identification rather than username. This is essentially the 80% case for 
> requests for custom User models anyway, and it provides an example of "how 
> to do it" for anyone that wants to follow the lead.
>
> I've also taken the opportunity to do a bit of refactoring of auth forms 
> and the AbstractUser to make re-use of common parts a bit easier.
>
> Code is available here. Other than bikeshed stuff (name of module, name of 
> classes etc) it shouldn't be too controversial:
>
> https://github.com/freakboy3742/django/tree/emailauth
>
> There are two things still pending on this patch:
>
>  1) Documentation. There isn't much to document, but obviously this is 
> required.
>
>  2) Tests that work. 
>
> It's the second point where I need some assistance (or at the very least, 
> some suggestions of directions I can go with this).
>
> There isn't that much that requires testing -- after all, it's a fairly 
> vanilla User model. A bit of light testing of forms, some checks to make 
> sure that the models can be created, and that they play well with 
> ModelBackend, and so on.
>
> However, there's a problem with including these tests as part of Django's 
> own testing infrastructure, due to clashes with Django's default User model.
>
> In a production app, you either want a normal Django User model, in which 
> case you don't include contrib.emailauth in your app; or you include 
> contrib.emailauth, and you set AUTH_USER_MODEL must be set (I haven't added 
> a formal check for this, but I could). As a result, there's no confusion 
> over which User model is active. I've done some manual testing, and the 
> user model all works as expected.
>
> While running Django's test suite, however, the suite installs both 
> contrib.auth and contrib.emailauth, but *doesn't* set AUTH_USER_MODEL to 
> emailauth -- it can't, because we need to test default User behaviours. 
>
> Both auth.User and emailauth.User have relations with Group, and the names 
> clash, so the test suite won't start. 
>
> If you deliberately silence the model integrity system checks, you get 
> other errors, caused by the fact that there's confusion over which 
> AUTH_USER_MODEL is active at any given time. 
>
> So - I'm looking for suggestions for how this can be handled in Django's 
> test suite. 
>
> I've thought about explicitly excluding contrib.emailauth from the test 
> suite's INSTALLED_APPS, but then the tests in emailauth don't get 
> discovered and executed. 
>
> Even if you reorganise the tests to make them part of the "django" tests, 
> rather than tests that are part of the contrib app, you get problems 
> because you end up with two models with the same relation name to Group.
>
> So - anyone got any ideas how to handle this? Is there some feature of the 
> new app loading code that I can exploit here?
>
> Yours,
> Russ Magee %-)
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a3259799-00a4-432c-ab59-d0401c44625e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: DisallowedHost causes 500 errors.

[Mark Lavin]

This was already changed in https://code.djangoproject.com/ticket/19866 
which is part of 1.6. It's noted under the minor 
features https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features

SuspiciousOperation has been differentiated into a number of subclasses, 
and each will log to a matching named logger under the django.security 
logging hierarchy. Along with this change, a handler400 mechanism and 
default view are used whenever a SuspiciousOperation reaches the WSGI 
handler to return an HttpResponseBadRequest.


Best,

Mark 

On Thursday, February 13, 2014 12:27:37 PM UTC-5, J. Cliff Dyer wrote:
>
> Django devs, 
>
> I've noticed that the new ALLOWED_HOSTS causes the site 
> to return HTTP-500 errors whenever a user submits a request with an 
> HTTP_HOST header that is not allowed.  This is flooding my developers' 
> inboxes with spurious error messages.  The site is refusing the 
> requests as expected, but it is doing so with a *server* error, when in 
> fact the error is a problem with what the *clients* are doing.  I'd 
> like to start a discussion here before I create a bug report, but I 
> believe this should instead be returning a 403 Forbidden response 
> (based on my reading of http://httpstatus.es/403), or at least 
> something in the 4xx (Client Error) range.  This might be appropriate 
> for other SuspiciousOperation exceptions as well, though it's probably 
> worth discussing just how they are each handled.   
>
> In short, I think 500 makes it look like there's something wrong 
> with the server, when in fact the response is being blocked because the 
> server is doing the right thing.  If there's some consensus that this 
> would be a helpful enhancement, I'll write up a bug report and see if I 
> can create a patch to accompany it. Otherwise, I'll try to fix it for 
> my own use in a custom middleware. 
>
> Cheers, 
> Cliff 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7344345d-691f-463a-805e-3f57ad58ad0c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Migrations and System Checks

[Mark Lavin]

I'll agree that it isn't a code issue like models or admin classes which
don't validate. However it is a project state problem and is something that
users would probably want to know about regardless of whether they are
using runserver. There are other workflow issues that could be helped by
the checks framework if it fits this case. Uncompiled .po files or using
the DB cache without creating the table are two core issues I can think of
off the top of my head. There are probably some workflow issues related to
static asset pipelining (ala django-compressor or django-pipeline) in the
community which could make use of this as well.

As far as not needing this for any other command I don't that's accurate.
You would probably want this warning for any command which might interact
with the unmigrated tables, shell is one and loaddata is another. If you
are missing a migration for the custom User table it could impact
createsuperuser or changepassword. Those are just the commands in
core/contrib. That doesn't include the numerous commands which exist in the
Django eco-system.

Best,

Mark



On Wed, Feb 12, 2014 at 6:56 PM, Andrew Godwin  wrote:

> So, I don't think it should work within that framework, as having
> unapplied migrations is not a _system issue_.
>
> It's only really a helpful reminder that fits into the most common part of
> the workflow (that being that you git pull, new migrations come in, and you
> run runserver to check out the site and things might start breaking).
> There's no need for it to run during any other command (in fact, the only
> one it might make sense with is shell). The checks framework is for
> whole-project warnings IMO (like "your foreignkeys clash" or "your tests
> are structured wrong"), and this just isn't that, it's a helpful workflow
> reminder that we could in fact remove without much problem.
>
> I'm not sure what a good description on the overall purpose of the checks
> is would be - others might be able to elaborate more - but from a practical
> point of view, I only want that code to run when you run runserver (much
> like when you run migrate we *also* run part of makemigrations if there's
> no migrations to run for another workflow prompt to see if you have
> changes), so I feel like it's sitting in an alright place for now.
>
> If we could get some context for the checks - things like "this check is
> only for interactive runserver sessions" - then it would fit there. Perhaps
> we do, I've not looked into checks too much.
>
> Andrew
>
>
> On Wed, Feb 12, 2014 at 10:28 PM, Mark Lavin  wrote:
>
>> Last weekend I set out to fix https://code.djangoproject.com/ticket/21856
>>
>> which was a 1.7 release blocker and holding up additional 1.7 testing for
>> a project
>> I'm currently working on. Upon diving in I saw that the checks for
>> un-applied migrations
>> were tied to the runserver command. I thought to myself that this
>> seemed more appropriate for the new system checks framework. I assumed
>> that it
>> hadn't been done this way originally because Andrew's work on migrations
>> was running
>> parallel to the system checks. I figured while I was fixing the issue at
>> hand it
>> would make sense to migrate (if you'll excuse the pun) this check to the
>> new framework.
>> After all the stated goal of these checks was to detecting common
>> problems
>> (such as un-applied migrations) and give hints on how to fix them (such
>> as "you should run migrate").
>> While the documentation for writing new checks was somewhat lacking, I
>> was able
>> to convert the existing check and test it rather cleanly
>> https://github.com/django/django/pull/2244
>>
>> However, this change created some issues I didn't foresee when writing or
>> testing
>> this through Django's test suite. In particular this warning for
>> un-applied migrations
>> was triggered when running the tests for a project and when running the
>> migrate command
>> itself. Obviously if one is running migrate to resolve the problem they
>> don't need
>> to be told to run migrate. It seemed to me that there would/should be a
>> way to suppress
>> these checks for these cases and the issue here was a shortcoming of the
>> system
>> check framework (or my understand of how to use it). Instead my work was
>> reverted
>> in favor of restoring the check only in the runserver command. Andrew's
>> comment in
>> the ticket seemed to indicate that I was misunderstanding the purpose of
>> the system
>> checks and that this check fo

Migrations and System Checks

[Mark Lavin]

Last weekend I set out to fix https://code.djangoproject.com/ticket/21856 
which was a 1.7 release blocker and holding up additional 1.7 testing for a 
project
I'm currently working on. Upon diving in I saw that the checks for 
un-applied migrations
were tied to the runserver command. I thought to myself that this
seemed more appropriate for the new system checks framework. I assumed that 
it
hadn't been done this way originally because Andrew's work on migrations 
was running
parallel to the system checks. I figured while I was fixing the issue at 
hand it
would make sense to migrate (if you'll excuse the pun) this check to the 
new framework. 
After all the stated goal of these checks was to detecting common problems 
(such as un-applied migrations) and give hints on how to fix them (such as 
"you should run migrate").
While the documentation for writing new checks was somewhat lacking, I was 
able
to convert the existing check and test it rather cleanly 
https://github.com/django/django/pull/2244

However, this change created some issues I didn't foresee when writing or 
testing
this through Django's test suite. In particular this warning for un-applied 
migrations
was triggered when running the tests for a project and when running the 
migrate command
itself. Obviously if one is running migrate to resolve the problem they 
don't need
to be told to run migrate. It seemed to me that there would/should be a way 
to suppress
these checks for these cases and the issue here was a shortcoming of the 
system
check framework (or my understand of how to use it). Instead my work was 
reverted 
in favor of restoring the check only in the runserver command. Andrew's 
comment in 
the ticket seemed to indicate that I was misunderstanding the purpose of 
the system 
checks and that this check for un-applied migrations did not fit into its 
purpose.

My questions for this list are:

Should the check for un-applied migrations work within the new system 
checks framework?

If so, how can we address this issue that in my original implementation 
these checks
are shown when they are clearly not needed such as when running migrate?

If not, can I get some more clarity on the purpose of these system checks?

Thanks,

Mark

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0cf1d803-7aa2-4342-9f1e-564a560f13ba%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: e-mail changed to email in email validator

[Mark Lavin]

This was changed as part of https://code.djangoproject.com/ticket/17899

Best,

Mark

On Wednesday, December 19, 2012 4:23:24 PM UTC-5, Skylar Saveland wrote:
>
> I checked out 1.5 branch today and found a change that I can't find the 
> debate/announcement for: e-mail to email in
>
> django/core/validators.py:validate_email = EmailValidator(email_re, 
> _('Enter a valid email address.'), 'invalid')
>
> This change has me with
>
> # in 1.4.x it is:
> #error = 'Enter a valid e-mail address.'
> # in 1.5.x it is:
> #error = 'Enter a valid email address.'
> #self.assertFormError(res, 'form', 'email', error)
>
> ❯ git checkout 1.4.3
> ❯ grep -r "Enter a valid email address." *
> (Nothing there)
> ~/Code/github/django (1f0af3c) ...
>
> > git checkout master
> ~/Code/github/django (master) ↑
> ❯ grep -r "Enter a valid email address." *
>  (lot's of stuff)
>
> I'm sure this was discussed and noted somewhere; but, where?
>
> -Skylar
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/llep9CQUXqwJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Trigger an event on add another

[Mark Lavin]

When I was working on a similar issue for django-selectable I created an 
issue to address this problem with a simple proof of concept patch 
https://code.djangoproject.com/ticket/15760. However I never followed 
through with the ticket to try to get something like this added. In the 
meantime another ticket was created and fixed which solves a large portion 
of the problem https://code.djangoproject.com/ticket/18241. With this 
change you can catch the click event of the add button. I still feel like 
the admin inline javascript is not particularly flexible and having some 
clearly documented hooks for this use case would be nice.

For what it's worth my solution in django-selectable was to patch the 
formset function to include my callback 
https://bitbucket.org/mlavin/django-selectable/src/efede3aa83ec6d372626eaac6f54c188287ec3d2/selectable/static/selectable/js/jquery.dj.selectable.js?at=default#cl-289
 Not 
the prettiest solution but it works.

Best,

Mark

On Friday, November 9, 2012 6:30:17 AM UTC-5, is_null wrote:
>
> Hi all,
>
> Is it possible to trigger an event on add-another ?
>
> Or is there a better solution to work around this problem ? 
> https://github.com/yourlabs/django-autocomplete-light/blob/master/autocomplete_light/static/autocomplete_light/widget.js#L267
>
> Thanks for answering
>
> Regards
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/zvzeX-CiNDgJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #3011 - Custom User Models -- Call for final review

[Mark Lavin]

I've been thinking about external app compatibility as well and it seems like:


from django.conf import settings


AUTH_USER_MODEL = getattr(settings, 'AUTH_USER_MODEL', 'auth.User')


class SomeModel(models.Model):

author = models.ForeignKey(AUTH_USER_MODEL)

is the easiest solution to me.


Best,


Mark


On Friday, September 28, 2012 8:14:57 AM UTC-4, rizumu wrote:
>
>
> It is great to see this merged, it has been a long time coming. :)
>
> I would like to add backwards compatibility to some apps and I'm looking 
> for a recommended technique. Could the following, or a better option, find 
> its way into the docs in a section specific to app developers tasked with 
> adding support for auth_user_model. Having a recommendation in the docs 
> would make pull requests with this feature more likely and easier to handle.
>
>
> try:
> from django.contrib.auth import get_user_model
> auth_user_model = get_user_model()
> except ImportError:
> from django.contrib.auth.models import User
> auth_user_model = User
>
> class SomeModel(models.Model):
>
> author = models.ForeignKey(auth_user_model)
>
> Because this isn't backwards compatible:
>
> class SomeModel(models.Model):
>
> author = models.ForeignKey(settings.AUTH_USER_MODEL)
>
> best,
> Tom
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/NukaEX99YXQJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Backporting some Python 3 support features to 1.4.x?

[Mark Lavin]

Hi,

I am one such author and I wanted to share my experience. I have ported my 
code and test suites for 3 applications to run on Django 1.3-1.5dev and 
Python 2.6, 2.7 and 3.2 (1.5dev only).
Those are https://github.com/mlavin/django-responsive, 
https://github.com/mlavin/django-ad-code and 
https://github.com/mlavin/django-scribbler. There are fairly new apps and 
don't
have many users/active deployments and so made good candidates for this 
work.

For the most part these were fairly easy to port. Adding unicode_literals 
everywhere was most of the work. There were a few places where I needed to 
make use of django.utils.six.
The largest pain point was trying to follow the recommendation for models 
__str__/__unicode__ in 
https://docs.djangoproject.com/en/dev/topics/python3/#str-and-unicode-methods
If you are trying to support Django 1.4 then you can't really use this 
recommendation without backporting the decorator because of the way it was 
written to be no-op for
Python 3 rather than Python 2. I understand why this decision was made and 
in the long run it is the right decision. For now I think app maintainers 
will have a bit of pain without
some of these features being backported to a 1.4.X release. You do have the 
option of using the hack you referenced (which I am doing) or a similar 
conditional import 
along with conditional __str__/__unicode__ which this decorator was meant 
to avoid. It's certainly not hard to do but it feels dirty.

My work now has been focused on getting the tests to pass and now they do. 
Another piece will be to actually deploy a project using one of these app.
I'll be continuing to do this work while at DjangoCon and I'll happily 
chime in any other stumbling blocks I run into while I port some other 
larger applications.

Best,

Mark

On Thursday, September 6, 2012 2:10:18 PM UTC-4, Aymeric Augustin wrote:
>
> Hello,
>
> Several people have started porting their apps for Python 3, but they're 
> running into trouble when they try to support both Django 1.4 and 
> master. They end up with regrettable patterns such as:
>
> try:
> from django.utils.six import PY3
> except ImportError:
> PY3 = False
>
> Authors of pluggable apps generally chose to support the current and the 
> previous version of Django. Thus their users can alternatively upgrade 
> Django and their apps, which makes upgrades less scary.
>
> Now we have two alternatives:
> 1) tell them to roll their own compat or wait until they can target 1.5 + 
> 1.6 — quite a downer,
> 2) add some forwards compatibility tools to 1.4.
>  
> We generally don't add features to minor releases, but there's a precedent 
> (csrf_noop), and the patch is as harmless as they come (see attachment).
>
> What do you think?
>  
> -- 
> Aymeric.
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/goU624tDqlkJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Don't assume that missing fields from POST data are equal to an empty string value.

[Mark Lavin]

I think Ian demonstrated exactly why this should not go in and that is not
all forms are ModelForms. Imagine writing a REST API which allows for an
optional format parameter for GET requests which is validated by a Django
form. With the inclusion of this proposal all clients would be forced to
specify the format in the GET (even if left blank) or else the form won't
validate. That's not much of an optional field and a large backwards
incompatibility.

On Fri, Jan 13, 2012 at 10:20 AM, Daniel Sokolowski <
daniel.sokolow...@klinsight.com> wrote:

> 1. How does this proposal effect default values specified on model fields?
> (ModelForm processing)
> 2. Forgive my ignorance but who has the final say if it goes in or not?
> And since it should be a yes what's next?
>
> On Thu, Jan 12, 2012 at 8:40 PM, Tai Lee  wrote:
>
>> Ian,
>>
>> I agree that there are a lot of different ways that form data can be
>> submitted to Django, including near endless combinations of multiple
>> Django forms, multiple HTML forms, AJAX, etc.
>>
>> If we reduce the scope of our discussion and consider only a Django
>> form (forgetting HTML forms and AJAX) and two possible scenarios:
>>
>> 1. Partial form data is bound to a Django form, and it is not expected
>> to result in a call to the form's `save()` method and a change to the
>> database. It is only intended to examine the form's validation state
>> relating to the partial data that is bound to the form. In this case,
>> the proposed change should have no impact.
>>
>> 2. Full form data is bound to a form, and it IS expected to validate
>> and result in a call to the form's `save()` method and a change to the
>> database. In this case, why would we ever want to assume that a
>> character field which has no bound data, is actually bound to an empty
>> string?
>>
>> This is a dangerous assumption, I think. If we intend to obtain
>> complete data from the user, bind it to a form and save it to the
>> database, we should be sure (as much as we can be) that the data is
>> actually complete.
>>
>> Take the following pure Django example:
>>
>> {{{
>> class Profile(models.Model):
>>first_name = models.CharField(blank=True, max_length=50)
>>last_name = models.CharField(blank=True, max_length=50)
>>address = models.CharField(blank=True, max_length=50)
>>
>> class ProfileForm(ModelForm):
>> class Meta:
>>model = Profile
>>
>> profile = Profile.objects.create(first_name='Tai', last_name='Lee',
>> address='Sydney')
>> form = ProfileForm({}, instance=profile)
>> if form.is_valid():
>>form.save()
>> }}}
>>
>> The profile will have no first name, last name or address. The form
>> will produce no validation errors and will save the updated model to
>> the database.
>>
>> This is very surprising and counter-intuitive to me.
>>
>> I think the only time we could safely make the assumption that fields
>> with empty string values will be omitted by the UA is when we are
>> confident that the source of the form data also makes a similar
>> assumption that the back-end will re-normalise those missing fields
>> back to an empty string.
>>
>> I'd be surprised if any UAs actually do make that assumption, because
>> the way Django treats missing character fields does not appear to be
>> based on any spec, and Django is not the only form processing back-end
>> (or even the most popular one).
>>
>> I'm not sure I follow your simplest case example of one HTML form in a
>> browser window and one Django Form in a view. If optional fields are
>> left off the HTML form deliberately, without change the Form class or
>> the view code, this is exactly when data loss will currently occur. I
>> think you are confusing optional as in "may not be specified by the
>> user" with optional as in "may not be processed by the form"?
>>
>> Cheers.
>> Tai.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers" group.
>> To post to this group, send email to django-developers@googlegroups.com.
>> To unsubscribe from this group, send email to
>> django-developers+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/django-developers?hl=en.
>>
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Don't assume that missing fields from POST data are equal to an empty string value.

[Mark Lavin]

> If optional fields are
> left off the HTML form deliberately, without change the Form class or
> the view code, this is exactly when data loss will currently occur.
> I think you are confusing optional as in "may not be specified by the
> user" with optional as in "may not be processed by the form"?

I think it would be more accurate to call this developer error rather
than a data loss bug in Django. If you are defining forms in this way
you are exposing all the model fields to be changed by the form. You
shouldn't be defining forms with fields you don't want changed.
Imagine this form:

{{{
from django import forms
from django.contrib.auth.models import User

class UserForm(forms.ModelForm):
 class Meta(object):
 model = User
}}}

Look innocent? It's not. Even if you don't render the is_staff or
is_superuser fields they can still be changed if they are given in the
POST. That means exposing this form can allow changing a user to a
superuser. Don't trust data from the client and be explicit about the
fields you want to expose for the form to change. This is why the
'fields' and 'exclude' Meta options exist.

Best,

Mark

On Jan 12, 9:40 pm, Tai Lee  wrote:
> Ian,
>
> I agree that there are a lot of different ways that form data can be
> submitted to Django, including near endless combinations of multiple
> Django forms, multiple HTML forms, AJAX, etc.
>
> If we reduce the scope of our discussion and consider only a Django
> form (forgetting HTML forms and AJAX) and two possible scenarios:
>
> 1. Partial form data is bound to a Django form, and it is not expected
> to result in a call to the form's `save()` method and a change to the
> database. It is only intended to examine the form's validation state
> relating to the partial data that is bound to the form. In this case,
> the proposed change should have no impact.
>
> 2. Full form data is bound to a form, and it IS expected to validate
> and result in a call to the form's `save()` method and a change to the
> database. In this case, why would we ever want to assume that a
> character field which has no bound data, is actually bound to an empty
> string?
>
> This is a dangerous assumption, I think. If we intend to obtain
> complete data from the user, bind it to a form and save it to the
> database, we should be sure (as much as we can be) that the data is
> actually complete.
>
> Take the following pure Django example:
>
> {{{
> class Profile(models.Model):
>     first_name = models.CharField(blank=True, max_length=50)
>     last_name = models.CharField(blank=True, max_length=50)
>     address = models.CharField(blank=True, max_length=50)
>
> class ProfileForm(ModelForm):
>     class Meta:
>         model = Profile
>
> profile = Profile.objects.create(first_name='Tai', last_name='Lee',
> address='Sydney')
> form = ProfileForm({}, instance=profile)
> if form.is_valid():
>     form.save()
>
> }}}
>
> The profile will have no first name, last name or address. The form
> will produce no validation errors and will save the updated model to
> the database.
>
> This is very surprising and counter-intuitive to me.
>
> I think the only time we could safely make the assumption that fields
> with empty string values will be omitted by the UA is when we are
> confident that the source of the form data also makes a similar
> assumption that the back-end will re-normalise those missing fields
> back to an empty string.
>
> I'd be surprised if any UAs actually do make that assumption, because
> the way Django treats missing character fields does not appear to be
> based on any spec, and Django is not the only form processing back-end
> (or even the most popular one).
>
> I'm not sure I follow your simplest case example of one HTML form in a
> browser window and one Django Form in a view. If optional fields are
> left off the HTML form deliberately, without change the Form class or
> the view code, this is exactly when data loss will currently occur. I
> think you are confusing optional as in "may not be specified by the
> user" with optional as in "may not be processed by the form"?
>
> Cheers.
> Tai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.