Re: Fix for Django Channels issue #962

[Carlton Gibson]

Hi Joonhyung.

Ah, the hardest issue in the box. Good. 
TBH I haven't had a chance to even look at that one yet, and I'll need to 
pull out the Stevens & Rago when I do...
— anyone here with serious chops, please do feel free to input. 

I think best if you comment on the issue. There's already a good discussion 
there. 

Also, there's really no harm in opening a PR to show the change. Even if 
it's not the right solution, it helps to have something to look at. 

Kind Regards,

Carlton



On Tuesday, 5 February 2019 13:15:55 UTC+1, joonhyu...@gmail.com wrote:
>
> Hello,
>
> This is my first post here. I'd like to discuss some ways of fixing the 
> issue for Django Channels.  
> Briefly 
> speaking, any test for Channels written with ChannelsLiveServerTestCase 
> freezes in MacOS.
>
> As a Mac user, I wanted to find a way to write tests, so I spent nearly 
> half a day debugging Channels, and finally got the answer. The thing was 
> that Python asyncio library does not behave very well with multiprocessing.
>
> To be specific, ChannelsLiveServerTestCase relies on DaphneProcess, which 
> again relies on asyncio. Finally, asyncio is based on Kqueue in BSD-like 
> OSes. Unfortunately, Kqueue is not inherited to a child process upon 
> fork(), while the file descriptor table does. This causes a problem when 
> the DaphneProcess of a child process tries to register event handlers for 
> sockets, because the file descriptor is still opened but the Kqueue does 
> not exist anymore. At this point, MacOS users see an error: bad file 
> descriptor. (I've checked lsof of the file descriptor, and found that it 
> points to /dev/null.)
>
> In Unix-like systems, although the above problem does not occur, the file 
> descriptor table is shared with its child, so forking while asyncio event 
> loop is opened should be avoided if possible. I suggest three ways to get 
> around this issue.
>
> 1. Close the forked loop and create a new event loop when DaphneProcess
>  runs.
> 2. Modify multiprocessing-based design to threading-based design. In other 
> words, run tests in single process, multi-thread environment.
> 3. Run tests in single process, single thread environment.
>
> The easiest way is 1, which I already committed in my own fork 
> ,
>  
> but this looks like a cheating, so I'm not very happy with this solution. 
> Nevertheless it passes all tests of Channels and Daphne, and moreover 
> Channels 
> tutorial  
> works fine with BSD-like OSes such as MacOS. (I've tested on both MacOS and 
> Linux.) If you think this is fine, then I will be happy to send a PR.
>
> Which one would be the most desirable design?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e8b58f70-f0ef-4b27-99ab-165c92643e2e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fix for Django Channels issue #962

[joonhyung . shin]

Hello,

This is my first post here. I'd like to discuss some ways of fixing the 
issue for Django Channels.  
Briefly 
speaking, any test for Channels written with ChannelsLiveServerTestCase 
freezes in MacOS.

As a Mac user, I wanted to find a way to write tests, so I spent nearly 
half a day debugging Channels, and finally got the answer. The thing was 
that Python asyncio library does not behave very well with multiprocessing.

To be specific, ChannelsLiveServerTestCase relies on DaphneProcess, which 
again relies on asyncio. Finally, asyncio is based on Kqueue in BSD-like 
OSes. Unfortunately, Kqueue is not inherited to a child process upon fork(), 
while the file descriptor table does. This causes a problem when the 
DaphneProcess of a child process tries to register event handlers for 
sockets, because the file descriptor is still opened but the Kqueue does 
not exist anymore. At this point, MacOS users see an error: bad file 
descriptor. (I've checked lsof of the file descriptor, and found that it 
points to /dev/null.)

In Unix-like systems, although the above problem does not occur, the file 
descriptor table is shared with its child, so forking while asyncio event 
loop is opened should be avoided if possible. I suggest three ways to get 
around this issue.

1. Close the forked loop and create a new event loop when DaphneProcess
 runs.
2. Modify multiprocessing-based design to threading-based design. In other 
words, run tests in single process, multi-thread environment.
3. Run tests in single process, single thread environment.

The easiest way is 1, which I already committed in my own fork 
,
 
but this looks like a cheating, so I'm not very happy with this solution. 
Nevertheless it passes all tests of Channels and Daphne, and moreover Channels 
tutorial  
works fine with BSD-like OSes such as MacOS. (I've tested on both MacOS and 
Linux.) If you think this is fine, then I will be happy to send a PR.

Which one would be the most desirable design?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/053d42af-e023-45bd-93fd-d5fc2998e7f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Extend FAQ with "How do I get Django and my JS framework to work together?"

[Jamesie Pic]

Throwing in some food for thought that may be worth mentioning and that we
like to play with:

- django-webpack-loader merged code chunking recently
- transcrypt allows coding ie with react in Python, transcrypt also
supports webpack
- chp is a project that aims to replace templates as we know them by
functions, it's a minimal port of react, it relies on transcrypt for
isomorphic support

>From my pov it seems webpack is establishing as a standard worth
supporting, but code chunking is really nice to have for a better loading
experience.

In pure Django we used to chunk by extending a script block in non-base
templates.

Le mar. 5 févr. 2019 à 11:17, Carlton Gibson  a
écrit :

> I think this topic is very interesting.
>
> Two sides of it:
>
> * Static files handling
> * APIs
>
> Curtis is right, there are other options but, Django REST Framework is
> (whilst not perfect) pretty solid on the API front. I think Django has a
> good story here.
> It's pretty hard not to find DRF if you follow any guide, or any searching
> at all.
>
> The static files story is a little different. It seems to me we don't tell
> the best story there.
>
> Rails has two things which we could be envious of, even if we didn't want
> to copy exactly:
>
> * The frontend framework integration that's already been mentioned.
> * The very easy "Ajax your form", with controllers (i.e. for us "generic
> views") automatically handling ajax form submissions.
>
> Both these features get users further quicker in these aspects than we are
> able to offer.
>
> We struggle to think of areas for improvements (re GSoC for example) but
> maybe here is an area.
>
> This ties into Claude's proposal here:
> https://groups.google.com/d/topic/django-developers/KYmNnvwXDUI/discussion
>
> My own story is, I've had lots of success with, and still use, Django
> Compressor.
>
>- At it's simplest you just map a content type to a shell command to
>run and then include your Sass/Less/React/Elm/whatever files in your HTML
>(with script or link tags, almost in the old-school way).
>- In development these are processed (& cached) per request.
>- For deployment you just run an offline compression task (management
>command) and then upload the files.
>- That's it.
>
> It's not a coverall approach — a frontend engineer will come along and
> totally replace Compressor with whatever is this week's Top Javascript
> Build System™ BUT it is a good 80:20: it lets me do something (that
> approximates) respectable, without knowing hardly anything about the latest
> frontend hotness. (GNU Make FTW! )
>
> I think if we were to offer something out-of-the-box that got as far as
> Compressor, or further, we'd:
>
>- satisfy most of our users,
>- allow yet more to get off the mark quickly,
>- and... well... those that need the full frontend toolchain would
>still be free to use it.
>
> I worry we'd never get anything like this into core... but I think it
> would be good. (As I say, I think it's one area where we are lacking/behind
> the competition.)
>
> Kind Regards,
>
> Carlton
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/49deee81-0230-48a0-8c2a-b12eb0956810%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op1_zPHFamYefkLPkYiRTnXjAbrX9wLa9XkQmYv5foNJKYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Extend FAQ with "How do I get Django and my JS framework to work together?"

[Carlton Gibson]

I think this topic is very interesting. 

Two sides of it: 

* Static files handling
* APIs

Curtis is right, there are other options but, Django REST Framework is 
(whilst not perfect) pretty solid on the API front. I think Django has a 
good story here. 
It's pretty hard not to find DRF if you follow any guide, or any searching 
at all. 

The static files story is a little different. It seems to me we don't tell 
the best story there. 

Rails has two things which we could be envious of, even if we didn't want 
to copy exactly:

* The frontend framework integration that's already been mentioned. 
* The very easy "Ajax your form", with controllers (i.e. for us "generic 
views") automatically handling ajax form submissions. 

Both these features get users further quicker in these aspects than we are 
able to offer. 

We struggle to think of areas for improvements (re GSoC for example) but 
maybe here is an area. 

This ties into Claude's proposal here: 
https://groups.google.com/d/topic/django-developers/KYmNnvwXDUI/discussion

My own story is, I've had lots of success with, and still use, Django 
Compressor.  

   - At it's simplest you just map a content type to a shell command to run 
   and then include your Sass/Less/React/Elm/whatever files in your HTML (with 
   script or link tags, almost in the old-school way). 
   - In development these are processed (& cached) per request. 
   - For deployment you just run an offline compression task (management 
   command) and then upload the files. 
   - That's it. 

It's not a coverall approach — a frontend engineer will come along and 
totally replace Compressor with whatever is this week's Top Javascript 
Build System™ BUT it is a good 80:20: it lets me do something (that 
approximates) respectable, without knowing hardly anything about the latest 
frontend hotness. (GNU Make FTW! ) 

I think if we were to offer something out-of-the-box that got as far as 
Compressor, or further, we'd:

   - satisfy most of our users, 
   - allow yet more to get off the mark quickly, 
   - and... well... those that need the full frontend toolchain would still 
   be free to use it. 
   
I worry we'd never get anything like this into core... but I think it would 
be good. (As I say, I think it's one area where we are lacking/behind the 
competition.)

Kind Regards,

Carlton

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/49deee81-0230-48a0-8c2a-b12eb0956810%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.