Re: A different approach for the auto-reloader
Buenas tardes David, cuentame El vie, 28 abr 2023 13:15, David Arredondo escribió: > Hola > > El vie, 28 de abr de 2023 12:02 p. m., natali...@gmail.com < > nataliabid...@gmail.com> escribió: > >> Hello everyone! >> >> I'm conducting a PR cleanse crusade for the project, where I'll try move >> forward those PRs that still make sense or close them when appropriate. >> Were there other developments in this topic that would be relevant to make >> a decision about the PR[0]? >> >> Thank you! >> Natalia. >> >> [0] https://github.com/django/django/pull/13154 >> >> On Wednesday, May 1, 2019 at 3:05:30 PM UTC-3 Adam Johnson wrote: >> >>> 1. It’s really hard to know where a request is going at this level, so we could end up with a page view triggering 20 static files requests very quickly, which would lead to stating all files 20x redundantly. >>> >>> >>> Indeed, maybe this could be solved with a re-stat cooldown? >>> >>> 2. How would we handle (potentially ridiculous) things like code imported from a thread during app ready? In general wouldn’t we need some kind of thread to poll for new imports to watch after Django has been started? >>> >>> >>> This is less ridiculous than it sounds, I'm working on an APM package >>> that does something like this. >>> >>> On Wed, 1 May 2019 at 18:38, Tom Forbes wrote: >>> Hey Ramiro, This sounds like a pretty awesome idea in general. It would also solve one long standing issue where the http socket is closed and re-opened during reloading, leading to pageviews after code changes failing. I’m trying to think of some edge cases where we need to be careful, and there are three I can think of: 1. It’s really hard to know where a request is going at this level, so we could end up with a page view triggering 20 static files requests very quickly, which would lead to stating all files 20x redundantly. 2. How would we handle (potentially ridiculous) things like code imported from a thread during app ready? In general wouldn’t we need some kind of thread to poll for new imports to watch after Django has been started? 3. Are there any workflows that would be disrupted by this? I can’t think of any that I use, but it’s possible that people have come to rely on the existing “does this throw an exception” in some cases, and triggering a request only to get this could be annoying? Overall though this feels like a good idea. Stat based reloaders are a lot simpler to deal with than platform specific fs monitoring. Ask thanks for the kind words about my refactor, I’m glad the abstractions work on something like this. Tom On 24 Apr 2019, at 05:33, Ramiro Morales wrote: Hi all, I had a stab at a somewhat simpler development server automatic reloading strategy https://github.com/django/django/compare/master...ramiro:synch-reloader Intention is to test how an implementation of a design by Gary Bernhardt would look. The best written description I could find is this: https://github.com/devlocker/tychus/issues/3 Gary also had posted some tweets (this is how I got interested in the topic) which seems to have been deleted since then. Main idea is: Actual checking of changes on the filesystem for modules under monitoring isn't performed in a loop or by depending on a OS kernel feature but per-HTTP request by a front-end proxy process which is in charge of restarting the 'upstream' web server process (in our case a dumbed-down runserver dev server) only when it detects there have been changes. Been meaning to try this for some time. It would have been much harder before Tom Forbes' work on refactoring and cleaning up the reloading code for Django 2.2. IMHO Tom's code is so very well thought that for example I just had to lightly subclass StatReload to implement this totally different strategy. Current form of the code is a new experimental 'serverrun' (for lack of a better name) added to the Django code base whose command line UI mimics 100% the runserver one. It copies code from a few places of our code base: The runserver command, the WSGI app hosting code, etc. I decided to implement this as a new built-in command for now a) to ease experimentation and b) because it needs some minor changes to the 'runserver' command to handle cosmetic details (logging). If the idea is accepted (read further below for reasons in favor of this) then maybe we can switch runserver to this code. Or if the idea isn't deemed appropate for Django core them I might implement it as an standalone django app/project. If the idea of a smarter stat()-based FS status monitor like this gets actually tested and validated in the field
Re: A different approach for the auto-reloader
Hola El vie, 28 de abr de 2023 12:02 p. m., natali...@gmail.com < nataliabid...@gmail.com> escribió: > Hello everyone! > > I'm conducting a PR cleanse crusade for the project, where I'll try move > forward those PRs that still make sense or close them when appropriate. > Were there other developments in this topic that would be relevant to make > a decision about the PR[0]? > > Thank you! > Natalia. > > [0] https://github.com/django/django/pull/13154 > > On Wednesday, May 1, 2019 at 3:05:30 PM UTC-3 Adam Johnson wrote: > >> 1. It’s really hard to know where a request is going at this level, so we >>> could end up with a page view triggering 20 static files requests very >>> quickly, which would lead to stating all files 20x redundantly. >> >> >> Indeed, maybe this could be solved with a re-stat cooldown? >> >> >>> 2. How would we handle (potentially ridiculous) things like code >>> imported from a thread during app ready? In general wouldn’t we need some >>> kind of thread to poll for new imports to watch after Django has been >>> started? >> >> >> This is less ridiculous than it sounds, I'm working on an APM package >> that does something like this. >> >> On Wed, 1 May 2019 at 18:38, Tom Forbes wrote: >> >>> Hey Ramiro, >>> This sounds like a pretty awesome idea in general. It would also solve >>> one long standing issue where the http socket is closed and re-opened >>> during reloading, leading to pageviews after code changes failing. >>> >>> I’m trying to think of some edge cases where we need to be careful, and >>> there are three I can think of: >>> 1. It’s really hard to know where a request is going at this level, so >>> we could end up with a page view triggering 20 static files requests very >>> quickly, which would lead to stating all files 20x redundantly. >>> >>> 2. How would we handle (potentially ridiculous) things like code >>> imported from a thread during app ready? In general wouldn’t we need some >>> kind of thread to poll for new imports to watch after Django has been >>> started? >>> >>> 3. Are there any workflows that would be disrupted by this? I can’t >>> think of any that I use, but it’s possible that people have come to rely on >>> the existing “does this throw an exception” in some cases, and triggering a >>> request only to get this could be annoying? >>> >>> Overall though this feels like a good idea. Stat based reloaders are a >>> lot simpler to deal with than platform specific fs monitoring. Ask thanks >>> for the kind words about my refactor, I’m glad the abstractions work on >>> something like this. >>> >>> Tom >>> >>> On 24 Apr 2019, at 05:33, Ramiro Morales wrote: >>> >>> Hi all, >>> >>> I had a stab at a somewhat simpler development server automatic >>> reloading strategy >>> https://github.com/django/django/compare/master...ramiro:synch-reloader >>> >>> Intention is to test how an implementation of a design by Gary Bernhardt >>> would look. The best written description I could find is this: >>> >>> https://github.com/devlocker/tychus/issues/3 >>> >>> Gary also had posted some tweets (this is how I got interested in the >>> topic) which seems to have been deleted since then. >>> >>> Main idea is: Actual checking of changes on the filesystem for modules >>> under monitoring isn't performed in a loop or by depending on a OS kernel >>> feature but per-HTTP request by a front-end proxy process which is in >>> charge of restarting the 'upstream' web server process (in our case a >>> dumbed-down runserver dev server) only when it detects there have been >>> changes. >>> >>> Been meaning to try this for some time. It would have been much harder >>> before Tom Forbes' work on refactoring and cleaning up the reloading code >>> for Django 2.2. IMHO Tom's code is so very well thought that for example I >>> just had to lightly subclass StatReload to implement this totally different >>> strategy. >>> >>> Current form of the code is a new experimental 'serverrun' (for lack of >>> a better name) added to the Django code base whose command line UI mimics >>> 100% the runserver one. >>> >>> It copies code from a few places of our code base: The runserver >>> command, the WSGI app hosting code, etc. >>> >>> I decided to implement this as a new built-in command for now a) to ease >>> experimentation and b) because it needs some minor changes to the >>> 'runserver' command to handle cosmetic details (logging). If the idea is >>> accepted (read further below for reasons in favor of this) then maybe we >>> can switch runserver to this code. Or if the idea isn't deemed appropate >>> for Django core them I might implement it as an standalone django >>> app/project. >>> >>> If the idea of a smarter stat()-based FS status monitor like this gets >>> actually tested and validated in the field (i.e. by users with big source >>> code trees) it could allow us to possibly stop needing to depend on all of: >>> >>> * watchman >>> * pyinotify >>> * watchdog >>> (and removing our support code for them
Re: A different approach for the auto-reloader
Hello everyone! I'm conducting a PR cleanse crusade for the project, where I'll try move forward those PRs that still make sense or close them when appropriate. Were there other developments in this topic that would be relevant to make a decision about the PR[0]? Thank you! Natalia. [0] https://github.com/django/django/pull/13154 On Wednesday, May 1, 2019 at 3:05:30 PM UTC-3 Adam Johnson wrote: > 1. It’s really hard to know where a request is going at this level, so we >> could end up with a page view triggering 20 static files requests very >> quickly, which would lead to stating all files 20x redundantly. > > > Indeed, maybe this could be solved with a re-stat cooldown? > > >> 2. How would we handle (potentially ridiculous) things like code imported >> from a thread during app ready? In general wouldn’t we need some kind of >> thread to poll for new imports to watch after Django has been started? > > > This is less ridiculous than it sounds, I'm working on an APM package that > does something like this. > > On Wed, 1 May 2019 at 18:38, Tom Forbes wrote: > >> Hey Ramiro, >> This sounds like a pretty awesome idea in general. It would also solve >> one long standing issue where the http socket is closed and re-opened >> during reloading, leading to pageviews after code changes failing. >> >> I’m trying to think of some edge cases where we need to be careful, and >> there are three I can think of: >> 1. It’s really hard to know where a request is going at this level, so we >> could end up with a page view triggering 20 static files requests very >> quickly, which would lead to stating all files 20x redundantly. >> >> 2. How would we handle (potentially ridiculous) things like code imported >> from a thread during app ready? In general wouldn’t we need some kind of >> thread to poll for new imports to watch after Django has been started? >> >> 3. Are there any workflows that would be disrupted by this? I can’t think >> of any that I use, but it’s possible that people have come to rely on the >> existing “does this throw an exception” in some cases, and triggering a >> request only to get this could be annoying? >> >> Overall though this feels like a good idea. Stat based reloaders are a >> lot simpler to deal with than platform specific fs monitoring. Ask thanks >> for the kind words about my refactor, I’m glad the abstractions work on >> something like this. >> >> Tom >> >> On 24 Apr 2019, at 05:33, Ramiro Morales wrote: >> >> Hi all, >> >> I had a stab at a somewhat simpler development server automatic reloading >> strategy >> https://github.com/django/django/compare/master...ramiro:synch-reloader >> >> Intention is to test how an implementation of a design by Gary Bernhardt >> would look. The best written description I could find is this: >> >> https://github.com/devlocker/tychus/issues/3 >> >> Gary also had posted some tweets (this is how I got interested in the >> topic) which seems to have been deleted since then. >> >> Main idea is: Actual checking of changes on the filesystem for modules >> under monitoring isn't performed in a loop or by depending on a OS kernel >> feature but per-HTTP request by a front-end proxy process which is in >> charge of restarting the 'upstream' web server process (in our case a >> dumbed-down runserver dev server) only when it detects there have been >> changes. >> >> Been meaning to try this for some time. It would have been much harder >> before Tom Forbes' work on refactoring and cleaning up the reloading code >> for Django 2.2. IMHO Tom's code is so very well thought that for example I >> just had to lightly subclass StatReload to implement this totally different >> strategy. >> >> Current form of the code is a new experimental 'serverrun' (for lack of a >> better name) added to the Django code base whose command line UI mimics >> 100% the runserver one. >> >> It copies code from a few places of our code base: The runserver command, >> the WSGI app hosting code, etc. >> >> I decided to implement this as a new built-in command for now a) to ease >> experimentation and b) because it needs some minor changes to the >> 'runserver' command to handle cosmetic details (logging). If the idea is >> accepted (read further below for reasons in favor of this) then maybe we >> can switch runserver to this code. Or if the idea isn't deemed appropate >> for Django core them I might implement it as an standalone django >> app/project. >> >> If the idea of a smarter stat()-based FS status monitor like this gets >> actually tested and validated in the field (i.e. by users with big source >> code trees) it could allow us to possibly stop needing to depend on all of: >> >> * watchman >> * pyinotify >> * watchdog >> (and removing our support code for them from the Django code base). >> >> Also, this would mean: >> >> * Setup simplification for final users (no third party Python libraries >> or system daemon to install) >> * Better
