Re: A different approach for the auto-reloader

[Salomon Roberto Herrera Navarro]

Buenas tardes David, cuentame

El vie, 28 abr 2023 13:15, David Arredondo 
escribió:

> Hola
>
> El vie, 28 de abr de 2023 12:02 p. m., natali...@gmail.com <
> nataliabid...@gmail.com> escribió:
>
>> Hello everyone!
>>
>> I'm conducting a PR cleanse crusade for the project, where I'll try move
>> forward those PRs that still make sense or close them when appropriate.
>> Were there other developments in this topic that would be relevant to make
>> a decision about the PR[0]?
>>
>> Thank you!
>> Natalia.
>>
>> [0] https://github.com/django/django/pull/13154
>>
>> On Wednesday, May 1, 2019 at 3:05:30 PM UTC-3 Adam Johnson wrote:
>>
>>> 1. It’s really hard to know where a request is going at this level, so
 we could end up with a page view triggering 20 static files requests very
 quickly, which would lead to stating all files 20x redundantly.
>>>
>>>
>>> Indeed, maybe this could be solved with a re-stat cooldown?
>>>
>>>
 2. How would we handle (potentially ridiculous) things like code
 imported from a thread during app ready? In general wouldn’t we need some
 kind of thread to poll for new imports to watch after Django has been
 started?
>>>
>>>
>>> This is less ridiculous than it sounds, I'm working on an APM package
>>> that does something like this.
>>>
>>> On Wed, 1 May 2019 at 18:38, Tom Forbes  wrote:
>>>
 Hey Ramiro,
 This sounds like a pretty awesome idea in general. It would also solve
 one long standing issue where the http socket is closed and re-opened
 during reloading, leading to pageviews after code changes failing.

 I’m trying to think of some edge cases where we need to be careful, and
 there are three I can think of:
 1. It’s really hard to know where a request is going at this level, so
 we could end up with a page view triggering 20 static files requests very
 quickly, which would lead to stating all files 20x redundantly.

 2. How would we handle (potentially ridiculous) things like code
 imported from a thread during app ready? In general wouldn’t we need some
 kind of thread to poll for new imports to watch after Django has been
 started?

 3. Are there any workflows that would be disrupted by this? I can’t
 think of any that I use, but it’s possible that people have come to rely on
 the existing “does this throw an exception” in some cases, and triggering a
 request only to get this could be annoying?

 Overall though this feels like a good idea. Stat based reloaders are a
 lot simpler to deal with than platform specific fs monitoring. Ask thanks
 for the kind words about my refactor, I’m glad the abstractions work on
 something like this.

 Tom

 On 24 Apr 2019, at 05:33, Ramiro Morales  wrote:

 Hi all,

 I had a stab at a somewhat simpler development server automatic
 reloading strategy
 https://github.com/django/django/compare/master...ramiro:synch-reloader

 Intention is to test how an implementation of a design by Gary
 Bernhardt would look. The best written description I could find is this:

 https://github.com/devlocker/tychus/issues/3

 Gary also had posted some tweets (this is how I got interested in the
 topic) which seems to have been deleted since then.

 Main idea is: Actual checking of changes on the filesystem for modules
 under monitoring isn't performed in a loop or by depending on a OS kernel
 feature but per-HTTP request by a front-end proxy process which is in
 charge of restarting the 'upstream' web server process (in our case a
 dumbed-down runserver dev server) only when it detects there have been
 changes.

 Been meaning to try this for some time. It would have been much harder
 before Tom Forbes' work on refactoring and cleaning up the reloading code
 for Django 2.2. IMHO Tom's code is so very well thought that for example I
 just had to lightly subclass StatReload to implement this totally different
 strategy.

 Current form of the code is a new experimental 'serverrun' (for lack of
 a better name) added to the Django code base whose command line UI mimics
 100% the runserver one.

 It copies code from a few places of our code base: The runserver
 command, the WSGI app hosting code, etc.

 I decided to implement this as a new built-in command for now a) to
 ease experimentation and b) because it needs some minor changes to the
 'runserver' command to handle cosmetic details (logging). If the idea is
 accepted (read further below for reasons in favor of this) then maybe we
 can switch runserver to this code. Or if the idea isn't deemed appropate
 for Django core them I might implement it as an standalone django
 app/project.

 If the idea of a smarter stat()-based FS status monitor like this gets
 actually tested and validated in the field 

Re: A different approach for the auto-reloader

[David Arredondo]

Hola

El vie, 28 de abr de 2023 12:02 p. m., natali...@gmail.com <
nataliabid...@gmail.com> escribió:

> Hello everyone!
>
> I'm conducting a PR cleanse crusade for the project, where I'll try move
> forward those PRs that still make sense or close them when appropriate.
> Were there other developments in this topic that would be relevant to make
> a decision about the PR[0]?
>
> Thank you!
> Natalia.
>
> [0] https://github.com/django/django/pull/13154
>
> On Wednesday, May 1, 2019 at 3:05:30 PM UTC-3 Adam Johnson wrote:
>
>> 1. It’s really hard to know where a request is going at this level, so we
>>> could end up with a page view triggering 20 static files requests very
>>> quickly, which would lead to stating all files 20x redundantly.
>>
>>
>> Indeed, maybe this could be solved with a re-stat cooldown?
>>
>>
>>> 2. How would we handle (potentially ridiculous) things like code
>>> imported from a thread during app ready? In general wouldn’t we need some
>>> kind of thread to poll for new imports to watch after Django has been
>>> started?
>>
>>
>> This is less ridiculous than it sounds, I'm working on an APM package
>> that does something like this.
>>
>> On Wed, 1 May 2019 at 18:38, Tom Forbes  wrote:
>>
>>> Hey Ramiro,
>>> This sounds like a pretty awesome idea in general. It would also solve
>>> one long standing issue where the http socket is closed and re-opened
>>> during reloading, leading to pageviews after code changes failing.
>>>
>>> I’m trying to think of some edge cases where we need to be careful, and
>>> there are three I can think of:
>>> 1. It’s really hard to know where a request is going at this level, so
>>> we could end up with a page view triggering 20 static files requests very
>>> quickly, which would lead to stating all files 20x redundantly.
>>>
>>> 2. How would we handle (potentially ridiculous) things like code
>>> imported from a thread during app ready? In general wouldn’t we need some
>>> kind of thread to poll for new imports to watch after Django has been
>>> started?
>>>
>>> 3. Are there any workflows that would be disrupted by this? I can’t
>>> think of any that I use, but it’s possible that people have come to rely on
>>> the existing “does this throw an exception” in some cases, and triggering a
>>> request only to get this could be annoying?
>>>
>>> Overall though this feels like a good idea. Stat based reloaders are a
>>> lot simpler to deal with than platform specific fs monitoring. Ask thanks
>>> for the kind words about my refactor, I’m glad the abstractions work on
>>> something like this.
>>>
>>> Tom
>>>
>>> On 24 Apr 2019, at 05:33, Ramiro Morales  wrote:
>>>
>>> Hi all,
>>>
>>> I had a stab at a somewhat simpler development server automatic
>>> reloading strategy
>>> https://github.com/django/django/compare/master...ramiro:synch-reloader
>>>
>>> Intention is to test how an implementation of a design by Gary Bernhardt
>>> would look. The best written description I could find is this:
>>>
>>> https://github.com/devlocker/tychus/issues/3
>>>
>>> Gary also had posted some tweets (this is how I got interested in the
>>> topic) which seems to have been deleted since then.
>>>
>>> Main idea is: Actual checking of changes on the filesystem for modules
>>> under monitoring isn't performed in a loop or by depending on a OS kernel
>>> feature but per-HTTP request by a front-end proxy process which is in
>>> charge of restarting the 'upstream' web server process (in our case a
>>> dumbed-down runserver dev server) only when it detects there have been
>>> changes.
>>>
>>> Been meaning to try this for some time. It would have been much harder
>>> before Tom Forbes' work on refactoring and cleaning up the reloading code
>>> for Django 2.2. IMHO Tom's code is so very well thought that for example I
>>> just had to lightly subclass StatReload to implement this totally different
>>> strategy.
>>>
>>> Current form of the code is a new experimental 'serverrun' (for lack of
>>> a better name) added to the Django code base whose command line UI mimics
>>> 100% the runserver one.
>>>
>>> It copies code from a few places of our code base: The runserver
>>> command, the WSGI app hosting code, etc.
>>>
>>> I decided to implement this as a new built-in command for now a) to ease
>>> experimentation and b) because it needs some minor changes to the
>>> 'runserver' command to handle cosmetic details (logging). If the idea is
>>> accepted (read further below for reasons in favor of this) then maybe we
>>> can switch runserver to this code. Or if the idea isn't deemed appropate
>>> for Django core them I might implement it as an standalone django
>>> app/project.
>>>
>>> If the idea of a smarter stat()-based FS status monitor like this gets
>>> actually tested and validated in the field (i.e. by users with big source
>>> code trees) it could allow us to possibly stop needing to depend on all of:
>>>
>>> * watchman
>>> * pyinotify
>>> * watchdog
>>> (and removing our support code for them 

Re: A different approach for the auto-reloader

[natali...@gmail.com]

Hello everyone!

I'm conducting a PR cleanse crusade for the project, where I'll try move 
forward those PRs that still make sense or close them when appropriate. 
Were there other developments in this topic that would be relevant to make 
a decision about the PR[0]?

Thank you!
Natalia.

[0] https://github.com/django/django/pull/13154

On Wednesday, May 1, 2019 at 3:05:30 PM UTC-3 Adam Johnson wrote:

> 1. It’s really hard to know where a request is going at this level, so we 
>> could end up with a page view triggering 20 static files requests very 
>> quickly, which would lead to stating all files 20x redundantly.
>
>
> Indeed, maybe this could be solved with a re-stat cooldown?
>  
>
>> 2. How would we handle (potentially ridiculous) things like code imported 
>> from a thread during app ready? In general wouldn’t we need some kind of 
>> thread to poll for new imports to watch after Django has been started?
>
>
> This is less ridiculous than it sounds, I'm working on an APM package that 
> does something like this.
>
> On Wed, 1 May 2019 at 18:38, Tom Forbes  wrote:
>
>> Hey Ramiro,
>> This sounds like a pretty awesome idea in general. It would also solve 
>> one long standing issue where the http socket is closed and re-opened 
>> during reloading, leading to pageviews after code changes failing.
>>
>> I’m trying to think of some edge cases where we need to be careful, and 
>> there are three I can think of:
>> 1. It’s really hard to know where a request is going at this level, so we 
>> could end up with a page view triggering 20 static files requests very 
>> quickly, which would lead to stating all files 20x redundantly.
>>
>> 2. How would we handle (potentially ridiculous) things like code imported 
>> from a thread during app ready? In general wouldn’t we need some kind of 
>> thread to poll for new imports to watch after Django has been started?
>>
>> 3. Are there any workflows that would be disrupted by this? I can’t think 
>> of any that I use, but it’s possible that people have come to rely on the 
>> existing “does this throw an exception” in some cases, and triggering a 
>> request only to get this could be annoying?
>>
>> Overall though this feels like a good idea. Stat based reloaders are a 
>> lot simpler to deal with than platform specific fs monitoring. Ask thanks 
>> for the kind words about my refactor, I’m glad the abstractions work on 
>> something like this.
>>
>> Tom
>>
>> On 24 Apr 2019, at 05:33, Ramiro Morales  wrote:
>>
>> Hi all,
>>
>> I had a stab at a somewhat simpler development server automatic reloading 
>> strategy 
>> https://github.com/django/django/compare/master...ramiro:synch-reloader
>>
>> Intention is to test how an implementation of a design by Gary Bernhardt 
>> would look. The best written description I could find is this:
>>
>> https://github.com/devlocker/tychus/issues/3
>>
>> Gary also had posted some tweets (this is how I got interested in the 
>> topic) which seems to have been deleted since then.
>>
>> Main idea is: Actual checking of changes on the filesystem for modules 
>> under monitoring isn't performed in a loop or by depending on a OS kernel 
>> feature but per-HTTP request by a front-end proxy process which is in 
>> charge of restarting the 'upstream' web server process (in our case a 
>> dumbed-down runserver dev server) only when it detects there have been 
>> changes. 
>>
>> Been meaning to try this for some time. It would have been much harder 
>> before Tom Forbes' work on refactoring and cleaning up the reloading code 
>> for Django 2.2. IMHO Tom's code is so very well thought that for example I 
>> just had to lightly subclass StatReload to implement this totally different 
>> strategy.
>>
>> Current form of the code is a new experimental 'serverrun' (for lack of a 
>> better name) added to the Django code base whose command line UI mimics 
>> 100% the runserver one. 
>>
>> It copies code from a few places of our code base: The runserver command, 
>> the WSGI app hosting code, etc.
>>
>> I decided to implement this as a new built-in command for now a) to ease 
>> experimentation and b) because it needs some minor changes to the 
>> 'runserver' command to handle cosmetic details (logging). If the idea is 
>> accepted (read further below for reasons in favor of this) then maybe we 
>> can switch runserver to this code. Or if the idea isn't deemed appropate 
>> for Django core them I might implement it as an standalone django 
>> app/project.
>>
>> If the idea of a smarter stat()-based FS status monitor like this gets 
>> actually tested and validated in the field (i.e. by users with big source 
>> code trees) it could allow us to possibly stop needing to depend on all of:
>>
>> * watchman
>> * pyinotify
>> * watchdog
>> (and removing our support code for them from the Django code base).
>>
>> Also, this would mean:
>>
>> * Setup simplification for final users (no third party Python libraries 
>> or system daemon to install)
>> * Better 

Re: A different approach for the auto-reloader

[Adam Johnson]

>
> 1. It’s really hard to know where a request is going at this level, so we
> could end up with a page view triggering 20 static files requests very
> quickly, which would lead to stating all files 20x redundantly.


Indeed, maybe this could be solved with a re-stat cooldown?


> 2. How would we handle (potentially ridiculous) things like code imported
> from a thread during app ready? In general wouldn’t we need some kind of
> thread to poll for new imports to watch after Django has been started?


This is less ridiculous than it sounds, I'm working on an APM package that
does something like this.

On Wed, 1 May 2019 at 18:38, Tom Forbes  wrote:

> Hey Ramiro,
> This sounds like a pretty awesome idea in general. It would also solve one
> long standing issue where the http socket is closed and re-opened during
> reloading, leading to pageviews after code changes failing.
>
> I’m trying to think of some edge cases where we need to be careful, and
> there are three I can think of:
> 1. It’s really hard to know where a request is going at this level, so we
> could end up with a page view triggering 20 static files requests very
> quickly, which would lead to stating all files 20x redundantly.
>
> 2. How would we handle (potentially ridiculous) things like code imported
> from a thread during app ready? In general wouldn’t we need some kind of
> thread to poll for new imports to watch after Django has been started?
>
> 3. Are there any workflows that would be disrupted by this? I can’t think
> of any that I use, but it’s possible that people have come to rely on the
> existing “does this throw an exception” in some cases, and triggering a
> request only to get this could be annoying?
>
> Overall though this feels like a good idea. Stat based reloaders are a lot
> simpler to deal with than platform specific fs monitoring. Ask thanks for
> the kind words about my refactor, I’m glad the abstractions work on
> something like this.
>
> Tom
>
> On 24 Apr 2019, at 05:33, Ramiro Morales  wrote:
>
> Hi all,
>
> I had a stab at a somewhat simpler development server automatic reloading
> strategy
> https://github.com/django/django/compare/master...ramiro:synch-reloader
>
> Intention is to test how an implementation of a design by Gary Bernhardt
> would look. The best written description I could find is this:
>
> https://github.com/devlocker/tychus/issues/3
>
> Gary also had posted some tweets (this is how I got interested in the
> topic) which seems to have been deleted since then.
>
> Main idea is: Actual checking of changes on the filesystem for modules
> under monitoring isn't performed in a loop or by depending on a OS kernel
> feature but per-HTTP request by a front-end proxy process which is in
> charge of restarting the 'upstream' web server process (in our case a
> dumbed-down runserver dev server) only when it detects there have been
> changes.
>
> Been meaning to try this for some time. It would have been much harder
> before Tom Forbes' work on refactoring and cleaning up the reloading code
> for Django 2.2. IMHO Tom's code is so very well thought that for example I
> just had to lightly subclass StatReload to implement this totally different
> strategy.
>
> Current form of the code is a new experimental 'serverrun' (for lack of a
> better name) added to the Django code base whose command line UI mimics
> 100% the runserver one.
>
> It copies code from a few places of our code base: The runserver command,
> the WSGI app hosting code, etc.
>
> I decided to implement this as a new built-in command for now a) to ease
> experimentation and b) because it needs some minor changes to the
> 'runserver' command to handle cosmetic details (logging). If the idea is
> accepted (read further below for reasons in favor of this) then maybe we
> can switch runserver to this code. Or if the idea isn't deemed appropate
> for Django core them I might implement it as an standalone django
> app/project.
>
> If the idea of a smarter stat()-based FS status monitor like this gets
> actually tested and validated in the field (i.e. by users with big source
> code trees) it could allow us to possibly stop needing to depend on all of:
>
> * watchman
> * pyinotify
> * watchdog
> (and removing our support code for them from the Django code base).
>
> Also, this would mean:
>
> * Setup simplification for final users (no third party Python libraries or
> system daemon to install)
> * Better cross-platform portability for Django (we go back to
> piggy-backing stat() from the stdlib as our only way yo trigger code
> reloading).
>
> Additionally, as the reloading is performed fully (by restarting the whole
> HTTP server) and is triggered from another process (the transparent http
> proxy one) we can drop some contortions we currently need to make:
>
> - Having to wait for the app registry stabilization
> - Avoiding race conditions with the url resolver
>
> I suspect there could be power efficiency advantages too as:
>
> * The scanning for 

Re: A different approach for the auto-reloader

[Tom Forbes]

Hey Ramiro,
This sounds like a pretty awesome idea in general. It would also solve one long 
standing issue where the http socket is closed and re-opened during reloading, 
leading to pageviews after code changes failing.

I’m trying to think of some edge cases where we need to be careful, and there 
are three I can think of:
1. It’s really hard to know where a request is going at this level, so we could 
end up with a page view triggering 20 static files requests very quickly, which 
would lead to stating all files 20x redundantly.

2. How would we handle (potentially ridiculous) things like code imported from 
a thread during app ready? In general wouldn’t we need some kind of thread to 
poll for new imports to watch after Django has been started?

3. Are there any workflows that would be disrupted by this? I can’t think of 
any that I use, but it’s possible that people have come to rely on the existing 
“does this throw an exception” in some cases, and triggering a request only to 
get this could be annoying?

Overall though this feels like a good idea. Stat based reloaders are a lot 
simpler to deal with than platform specific fs monitoring. Ask thanks for the 
kind words about my refactor, I’m glad the abstractions work on something like 
this.

Tom

> On 24 Apr 2019, at 05:33, Ramiro Morales  wrote:
> 
> Hi all,
> 
> I had a stab at a somewhat simpler development server automatic reloading 
> strategy 
> https://github.com/django/django/compare/master...ramiro:synch-reloader
> 
> Intention is to test how an implementation of a design by Gary Bernhardt 
> would look. The best written description I could find is this:
> 
> https://github.com/devlocker/tychus/issues/3
> 
> Gary also had posted some tweets (this is how I got interested in the topic) 
> which seems to have been deleted since then.
> 
> Main idea is: Actual checking of changes on the filesystem for modules under 
> monitoring isn't performed in a loop or by depending on a OS kernel feature 
> but per-HTTP request by a front-end proxy process which is in charge of 
> restarting the 'upstream' web server process (in our case a dumbed-down 
> runserver dev server) only when it detects there have been changes. 
> 
> Been meaning to try this for some time. It would  have been much harder 
> before Tom Forbes' work on refactoring and cleaning up the reloading code for 
> Django 2.2. IMHO Tom's code is so very well thought that for example I just 
> had to lightly subclass StatReload to implement this totally different 
> strategy.
> 
> Current form of the code is a new experimental 'serverrun' (for lack of a 
> better name) added to the Django code base whose command line UI mimics 100% 
> the runserver one. 
> 
> It copies code from a few places of our code base: The runserver command, the 
> WSGI app hosting code, etc.
> 
> I decided to implement this as a new built-in command for now a) to ease 
> experimentation and b) because it needs some minor changes to the 'runserver' 
> command to handle cosmetic details (logging). If the idea is accepted (read 
> further below for reasons in favor of this) then maybe we can switch 
> runserver to this code. Or if the idea isn't deemed appropate for Django core 
> them I might implement it as an standalone  django app/project.
> 
> If the idea of a smarter stat()-based FS status monitor like this gets 
> actually tested and validated in the field (i.e. by users with big source 
> code trees) it could allow us to possibly stop needing to depend on all of:
> 
> * watchman
> * pyinotify
> * watchdog
> (and removing our support code for them from the Django code base).
> 
> Also, this would mean:
> 
> * Setup simplification for final users (no third party Python libraries or 
> system daemon to install)
> * Better cross-platform portability for Django (we go back to piggy-backing 
> stat() from the stdlib as our only way yo trigger code reloading).
> 
> Additionally, as the reloading is performed fully (by restarting the whole 
> HTTP server) and is triggered from another process (the transparent http 
> proxy one) we can drop some contortions we currently need to make:
> 
> - Having to wait for the app registry stabilization
> - Avoiding race conditions with the url resolver
> 
> I suspect there could be power efficiency advantages too as:
> 
> * The scanning for changes is triggered by HTTP requests which should be less 
> frequent than periodically every N seconds.
> * If the developer modifies more than one file before switching to the 
> browser there is need of only one FS scan to cater for all these changes, 
> which is performed just in time for the first HTTP request so the code 
> executed to render/serve it is 100% accurate in regard to actually reflecting 
> the state of the code on disk.
> 
> Similar projects include:
> - serveit: https://github.com/garybernhardt/serveit
> - tychus: https://github.com/devlocker/tychus
> - wsgiwatch: https://github.com/dpk/wsgiwatch
> 
> Feedback is welcome!
> 

Re: A different approach for the auto-reloader

[Andrew Godwin]

>From my read this also looks like it would make the auto-reloader able to
work a lot better with an async-capable server, so I would be in favour
given that is likely in the future as well.

Andrew

On Tue, Apr 23, 2019 at 9:33 PM Ramiro Morales  wrote:

> Hi all,
>
> I had a stab at a somewhat simpler development server automatic reloading
> strategy
> https://github.com/django/django/compare/master...ramiro:synch-reloader
>
> Intention is to test how an implementation of a design by Gary Bernhardt
> would look. The best written description I could find is this:
>
> https://github.com/devlocker/tychus/issues/3
>
> Gary also had posted some tweets (this is how I got interested in the
> topic) which seems to have been deleted since then.
>
> Main idea is: Actual checking of changes on the filesystem for modules
> under monitoring isn't performed in a loop or by depending on a OS kernel
> feature but per-HTTP request by a front-end proxy process which is in
> charge of restarting the 'upstream' web server process (in our case a
> dumbed-down runserver dev server) only when it detects there have been
> changes.
>
> Been meaning to try this for some time. It would have been much harder
> before Tom Forbes' work on refactoring and cleaning up the reloading code
> for Django 2.2. IMHO Tom's code is so very well thought that for example I
> just had to lightly subclass StatReload to implement this totally different
> strategy.
>
> Current form of the code is a new experimental 'serverrun' (for lack of a
> better name) added to the Django code base whose command line UI mimics
> 100% the runserver one.
>
> It copies code from a few places of our code base: The runserver command,
> the WSGI app hosting code, etc.
>
> I decided to implement this as a new built-in command for now a) to ease
> experimentation and b) because it needs some minor changes to the
> 'runserver' command to handle cosmetic details (logging). If the idea is
> accepted (read further below for reasons in favor of this) then maybe we
> can switch runserver to this code. Or if the idea isn't deemed appropate
> for Django core them I might implement it as an standalone django
> app/project.
>
> If the idea of a smarter stat()-based FS status monitor like this gets
> actually tested and validated in the field (i.e. by users with big source
> code trees) it could allow us to possibly stop needing to depend on all of:
>
> * watchman
> * pyinotify
> * watchdog
> (and removing our support code for them from the Django code base).
>
> Also, this would mean:
>
> * Setup simplification for final users (no third party Python libraries or
> system daemon to install)
> * Better cross-platform portability for Django (we go back to
> piggy-backing stat() from the stdlib as our only way yo trigger code
> reloading).
>
> Additionally, as the reloading is performed fully (by restarting the whole
> HTTP server) and is triggered from another process (the transparent http
> proxy one) we can drop some contortions we currently need to make:
>
> - Having to wait for the app registry stabilization
> - Avoiding race conditions with the url resolver
>
> I suspect there could be power efficiency advantages too as:
>
> * The scanning for changes is triggered by HTTP requests which should be
> less frequent than periodically every N seconds.
> * If the developer modifies more than one file before switching to the
> browser there is need of only one FS scan to cater for all these changes,
> which is performed just in time for the first HTTP request so the code
> executed to render/serve it is 100% accurate in regard to actually
> reflecting the state of the code on disk.
>
> Similar projects include:
> - serveit: https://github.com/garybernhardt/serveit
> - tychus: https://github.com/devlocker/tychus
> - wsgiwatch: https://github.com/dpk/wsgiwatch
>
> Feedback is welcome!
>
> Regards,
>
> --
> Ramiro Morales
> @ramiromorales
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAO7PdF99hUobeXs8JQyb%3DywDJ6bkkKWyhUYC%3DEa9JzwQM%2BH_5Q%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: A different approach for the auto-reloader

[Carlton Gibson]

Hi Ramiro. 

I had a quick look at this — it looks great. (And various folks are having 
fun with Watchman, so if the promise delivers it'll be welcome.) 

Do you want to make an actual PR (perhaps with a ticket) so we can get a 
proper review going? 

Thanks for the input, as ever! 

Kind Regards,

Carlton


On Wednesday, 24 April 2019 06:33:50 UTC+2, Ramiro Morales wrote:
>
> Hi all,
>
> I had a stab at a somewhat simpler development server automatic reloading 
> strategy 
> https://github.com/django/django/compare/master...ramiro:synch-reloader
>
> Intention is to test how an implementation of a design by Gary Bernhardt 
> would look. The best written description I could find is this:
>
> https://github.com/devlocker/tychus/issues/3
>
> Gary also had posted some tweets (this is how I got interested in the 
> topic) which seems to have been deleted since then.
>
> Main idea is: Actual checking of changes on the filesystem for modules 
> under monitoring isn't performed in a loop or by depending on a OS kernel 
> feature but per-HTTP request by a front-end proxy process which is in 
> charge of restarting the 'upstream' web server process (in our case a 
> dumbed-down runserver dev server) only when it detects there have been 
> changes. 
>
> Been meaning to try this for some time. It would have been much harder 
> before Tom Forbes' work on refactoring and cleaning up the reloading code 
> for Django 2.2. IMHO Tom's code is so very well thought that for example I 
> just had to lightly subclass StatReload to implement this totally different 
> strategy.
>
> Current form of the code is a new experimental 'serverrun' (for lack of a 
> better name) added to the Django code base whose command line UI mimics 
> 100% the runserver one. 
>
> It copies code from a few places of our code base: The runserver command, 
> the WSGI app hosting code, etc.
>
> I decided to implement this as a new built-in command for now a) to ease 
> experimentation and b) because it needs some minor changes to the 
> 'runserver' command to handle cosmetic details (logging). If the idea is 
> accepted (read further below for reasons in favor of this) then maybe we 
> can switch runserver to this code. Or if the idea isn't deemed appropate 
> for Django core them I might implement it as an standalone django 
> app/project.
>
> If the idea of a smarter stat()-based FS status monitor like this gets 
> actually tested and validated in the field (i.e. by users with big source 
> code trees) it could allow us to possibly stop needing to depend on all of:
>
> * watchman
> * pyinotify
> * watchdog
> (and removing our support code for them from the Django code base).
>
> Also, this would mean:
>
> * Setup simplification for final users (no third party Python libraries or 
> system daemon to install)
> * Better cross-platform portability for Django (we go back to 
> piggy-backing stat() from the stdlib as our only way yo trigger code 
> reloading).
>
> Additionally, as the reloading is performed fully (by restarting the whole 
> HTTP server) and is triggered from another process (the transparent http 
> proxy one) we can drop some contortions we currently need to make:
>
> - Having to wait for the app registry stabilization
> - Avoiding race conditions with the url resolver
>
> I suspect there could be power efficiency advantages too as:
>
> * The scanning for changes is triggered by HTTP requests which should be 
> less frequent than periodically every N seconds.
> * If the developer modifies more than one file before switching to the 
> browser there is need of only one FS scan to cater for all these changes, 
> which is performed just in time for the first HTTP request so the code 
> executed to render/serve it is 100% accurate in regard to actually 
> reflecting the state of the code on disk.
>
> Similar projects include:
> - serveit: https://github.com/garybernhardt/serveit
> - tychus: https://github.com/devlocker/tychus
> - wsgiwatch: https://github.com/dpk/wsgiwatch
>
> Feedback is welcome!
>
> Regards,
>
> -- 
> Ramiro Morales
> @ramiromorales
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/edf0baf4-f6d0-468d-86e3-c4f06d5e82d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

A different approach for the auto-reloader

[Ramiro Morales]

Hi all,

I had a stab at a somewhat simpler development server automatic reloading
strategy
https://github.com/django/django/compare/master...ramiro:synch-reloader

Intention is to test how an implementation of a design by Gary Bernhardt
would look. The best written description I could find is this:

https://github.com/devlocker/tychus/issues/3

Gary also had posted some tweets (this is how I got interested in the
topic) which seems to have been deleted since then.

Main idea is: Actual checking of changes on the filesystem for modules
under monitoring isn't performed in a loop or by depending on a OS kernel
feature but per-HTTP request by a front-end proxy process which is in
charge of restarting the 'upstream' web server process (in our case a
dumbed-down runserver dev server) only when it detects there have been
changes.

Been meaning to try this for some time. It would have been much harder
before Tom Forbes' work on refactoring and cleaning up the reloading code
for Django 2.2. IMHO Tom's code is so very well thought that for example I
just had to lightly subclass StatReload to implement this totally different
strategy.

Current form of the code is a new experimental 'serverrun' (for lack of a
better name) added to the Django code base whose command line UI mimics
100% the runserver one.

It copies code from a few places of our code base: The runserver command,
the WSGI app hosting code, etc.

I decided to implement this as a new built-in command for now a) to ease
experimentation and b) because it needs some minor changes to the
'runserver' command to handle cosmetic details (logging). If the idea is
accepted (read further below for reasons in favor of this) then maybe we
can switch runserver to this code. Or if the idea isn't deemed appropate
for Django core them I might implement it as an standalone django
app/project.

If the idea of a smarter stat()-based FS status monitor like this gets
actually tested and validated in the field (i.e. by users with big source
code trees) it could allow us to possibly stop needing to depend on all of:

* watchman
* pyinotify
* watchdog
(and removing our support code for them from the Django code base).

Also, this would mean:

* Setup simplification for final users (no third party Python libraries or
system daemon to install)
* Better cross-platform portability for Django (we go back to piggy-backing
stat() from the stdlib as our only way yo trigger code reloading).

Additionally, as the reloading is performed fully (by restarting the whole
HTTP server) and is triggered from another process (the transparent http
proxy one) we can drop some contortions we currently need to make:

- Having to wait for the app registry stabilization
- Avoiding race conditions with the url resolver

I suspect there could be power efficiency advantages too as:

* The scanning for changes is triggered by HTTP requests which should be
less frequent than periodically every N seconds.
* If the developer modifies more than one file before switching to the
browser there is need of only one FS scan to cater for all these changes,
which is performed just in time for the first HTTP request so the code
executed to render/serve it is 100% accurate in regard to actually
reflecting the state of the code on disk.

Similar projects include:
- serveit: https://github.com/garybernhardt/serveit
- tychus: https://github.com/devlocker/tychus
- wsgiwatch: https://github.com/dpk/wsgiwatch

Feedback is welcome!

Regards,

-- 
Ramiro Morales
@ramiromorales

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAO7PdF99hUobeXs8JQyb%3DywDJ6bkkKWyhUYC%3DEa9JzwQM%2BH_5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.