Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mariusz Felisiak ):

 In [changeset:"c00bc27945d195295101c695e2a901b06b10fe96" c00bc279]:
 {{{
 #!CommitTicketReference repository=""
 revision="c00bc27945d195295101c695e2a901b06b10fe96"
 Refs #30372 -- Stopped watching built-in Django translation files by auto-
 reloader.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.fdf374ad116ffce0591abab19180b26a%40djangoproject.com.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Andy Terra):

 Replying to [comment:15 Tom Forbes]:
 > Hey Andy,
 >
 > I have a hunch about where some of this excess load comes from. If
 you're willing it would be very helpful if you could try installing my
 branch:
 >
 > {{{
 > pip install https://github.com/orf/django/archive/30372-speedups.zip
 > }}}
 >
 > Alternatively you could set the `USE_I18N` setting to `False` in your
 project.
 >
 > I think it's the code that recurses through the `locale` directories to
 watch for changes to `.mo` files. Django ships with a large number of
 built in translation files that never change, and iterating through the
 several hundred directories might be the cause of some of the slowdown
 you're seeing.

 Tom,

 Just wanted to say I saw your comment yesterday and will look into it. I'm
 having trouble replicating the high CPU load now, even though it was
 consistent for *months*... will let you know once I find anything more
 definitive.


 Best,
 Andy

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.dd1ebd5b09ca15bec1c0c26d02123646%40djangoproject.com.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Tom Forbes):

 Hey Andy,

 I have a hunch about where some of this excess load comes from. If you're
 willing it would be very helpful if you could try installing my branch:

 {{{
 pip install https://github.com/orf/django/archive/30372-speedups.zip
 }}}

 Alternatively you could set the `USE_I18N` setting to `False` in your
 project.

 I think it's the code that recurses through the `locale` directories to
 watch for changes to `.mo` files. Django ships with a large number of
 built in translation files that never change, and iterating through the
 several hundred directories might be the cause of some of the slowdown
 you're seeing.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.24fad14355760c833f35acbb98eb8a1b%40djangoproject.com.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Tom Forbes):

 * cc: Tom Forbes (added)


Comment:

 Hey Benjamin,
 It seems some dependencies you have do not work with Django master. That's
 really annoying.

 I really hope the underlying issue is fixed in the new point release.
 Django 2.2.1 has been released, can you please test with that and let us
 know if it still has elevated CPU usage?

 We've also increased the quality and amount of logs that are produced with
 the autoreloader, if you could configure the `django.utils.autoreload`
 logger to be 'DEBUG' and attach the logs here it would be increadibly
 helpful in diagnosing this issue.

 Regarding the watchman memory usage, that may not be completely accurate
 representation of the total 'actual' memory being used. Regardless, for
 such a small project the CPU usage for a StatReloader should be very, very
 low.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.fd08189a9bc68fe754bfdfd7168f27a9%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Benjamin Schollnick):

 Seeing the same results with 2.2.1.

 As I mentioned before, I have no issues with testing, but I was unable to
 get the Dev GitHub branch to work properly...

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.9c15f98ad099920f4cc8074c90750497%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Benjamin Schollnick):

 To clarify, Django-3.0.dev20190430142051 is what was fetched from Git..

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.ce8bf6527c2779b4a81b222818c1bfee%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Benjamin Schollnick):

 Okay, I'm having issues with get the dev version to work...

 I followed the dev install instructions here -
 https://docs.djangoproject.com/en/2.2/topics/install/

 And I'm running into all sorts of ImportErrors, for example:

   File "/Users/benjamin/venvs/quickbbs/lib/python3.7/site-
 packages/allauth/account/models.py", line 9, in 
 from django.utils.encoding import python_2_unicode_compatible
 ImportError: cannot import name 'python_2_unicode_compatible' from
 'django.utils.encoding'
 (/Volumes/4TB_Drive/gallery/quickbbs/django/django/utils/encoding.py)

 So can you clarify that I am properly setting up the dev environment?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.76de2ad126a60234cfe74d032fe3f77a%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Carlton Gibson):

 * status:  new => closed
 * resolution:   => needsinfo


Comment:

 I'm going to close this as **needsinfo** for now. If it comes up for other
 people or we get more details we can reopen but it's kind of in a no-man's
 land at the moment.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.2f68d2b8869c9a548b0810719e905043%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Carlton Gibson):

 Hi Benjamin,

 First off, can you try against the latest master? There will be some fixes
 for the autoloader going into v2.2.1.
 (See 6754bffa2b2df15a741008aa611c1bb0e8dff22b)

 Then, can you help us pin down a bit more? Is this after extended usage
 etc? (i.e. implying some kind of leak...) And so on.
 Testing locally, it looks OK, so I'm wondering what's happening in your
 case.

 I don't want to just close as `needsinfo` if there is an issue, but, kind
 of "Needs info" at this stage. Maybe asking if others have seen similar
 would help?

 Thanks

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.bf68a98c7eea6921b2caa5d4d698147c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Benjamin Schollnick):

 Oh, and I just noticed, that watchman takes over 1.3 GB of memory!?!?!?

 System check identified 1 issue (0 silenced).
 April 21, 2019 - 20:33:39
 Django version 2.2, using settings 'quickbbs.settings'
 Starting development server at http://0.0.0.0:/
 Quit the server with CONTROL-C.
 Error connecting to Watchman: timed out waiting for response, while
 executing ('watch-project', '/Volumes/4TB_Drive/gallery/quickbbs')
 Watching for file changes with StatReloader

  watchman   ''**1.27 GB**'' 32  89  68350   Benjamin
 0.0 33.00   -   No  No  0 bytes 0 bytes 0
 0   0 bytes 0 bytes 64 bit  0 bytes 0 bytes 0 bytes 0 bytes No  No
 0 bytes No

 I'm sorry, but that's a insane amount of memory for a project that takes
 9.8Mb on disk.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.e098ef6f767f606012f958661ef20f4e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Benjamin Schollnick):

 Hate to say this, but can you elaborate on the django.utils.autoreload
 setting to log debug level?

 Well, I fought my way through setting up watchman, since performing

 from django.utils import autoreload
 autoreload.WatchmanReloader.check_availability()

 Was indicating that Watchman wasn't running or was incorrectly
 configured...

 But:

 1) Ensuring Watchman is running correctly, wasn't easy, the documentation
 for Watchman isn't too fantastic
 2) Watchman at defecto "idle" is eating 2-4% of the cpu
 3) With Watchman monitoring on, I saw no significant decrease in CPU
 usage.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.2cb5eed37d753cd72cdf6818cf7b74db%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Tomer Chachamu):

 Maybe configure the level of the 'django.utils.autoreload' logger to DEBUG
 and paste some logs here? Do you get "Watching for file changes with
 StatReloader" or "Watching for file changes with WatchmanReloader"?

 You can also run the manage.py shell and run this:

 from django.utils import autoreload
 autoreload.WatchmanReloader.check_availability()

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.fdca771b4f989677eb42106eaea1ada3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Benjamin Schollnick):

 I see no difference in the cpu utilization if I install pywatchman /
 watchman in MOSX.  The only resolution that I have found is to regress to
 2.1.8, which drops the CPU usage down to ~1%.

 I'm happy to run any other additional diagnostic suggestions that you
 have...  I'm just unclear on the next steps

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.447543f76b2e631d67f57c336d200d2f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Tobias Kunze):

 * cc: Tobias Kunze (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.9ba786668d6050ad3f756a03d92f8bd0%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Tobias Kunze):

 Does this happen independently of the size of your project? Does it change
 if you install `pywatchman` and `watchman` (with Django 2.2)?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.3e3279dc89e66b40a3693cd02f69ce7d%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #30372: Django (moderately) High CPU usage at Idle

[Django]

#30372: Django (moderately) High CPU usage at Idle
-+-
 Reporter:  Benjamin Schollnick  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Other) |  Version:  2.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Benjamin Schollnick):

 The processor load on Windows 10 (64bit) Pro, is slightly lower.
 Averaging ~8-10% w/Win 10 v1809, i5-6300u @ 2.40 Ghz w/8 Gb of ram.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/069.b7ebd1ad7793f3e4d3a99a6067ce9186%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.