[issue44399] log rotator cookbook example might waste disk space

[Vinay Sajip]

Change by Vinay Sajip :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[Vinay Sajip]

Vinay Sajip  added the comment:


New changeset 174fbd85f60d7877d3a4f58214a852209f8dfbea by Vinay Sajip in branch 
'main':
bpo-44399: Document another example of unexpected behavior. (GH-27407)
https://github.com/python/cpython/commit/174fbd85f60d7877d3a4f58214a852209f8dfbea


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[Vinay Sajip]

Change by Vinay Sajip :


--
pull_requests: +25940
pull_request: https://github.com/python/cpython/pull/27407

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[Vinay Sajip]

Vinay Sajip  added the comment:


New changeset 5a0c6abb569cadfc6b418a7d23ecffec01bd234d by Miss Islington (bot) 
in branch '3.9':
[3.9] bpo-44399: Update logging cookbook to document patterns to be avoided. 
(GH-27348) (GH-27350)
https://github.com/python/cpython/commit/5a0c6abb569cadfc6b418a7d23ecffec01bd234d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[Vinay Sajip]

Vinay Sajip  added the comment:


New changeset 628baf6fef4d633e756024cf813941a4075360c1 by Miss Islington (bot) 
in branch '3.10':
[3.10] bpo-44399: Update logging cookbook to document patterns to be avoided. 
(GH-27348) (GH-27352)
https://github.com/python/cpython/commit/628baf6fef4d633e756024cf813941a4075360c1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[miss-islington]

Change by miss-islington :


--
pull_requests: +25892
pull_request: https://github.com/python/cpython/pull/27351

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[miss-islington]

Change by miss-islington :


--
pull_requests: +25893
pull_request: https://github.com/python/cpython/pull/27352

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 3.0 -> 4.0
pull_requests: +25891
pull_request: https://github.com/python/cpython/pull/27350

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[Vinay Sajip]

Vinay Sajip  added the comment:


New changeset 9751f85914e0ef3324671a91da34a635d48b17fb by Vinay Sajip in branch 
'main':
bpo-44399: Update logging cookbook to document patterns to be avoided. 
(GH-27348)
https://github.com/python/cpython/commit/9751f85914e0ef3324671a91da34a635d48b17fb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[Vinay Sajip]

Change by Vinay Sajip :


--
keywords: +patch
pull_requests: +25889
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/27348

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[Vinay Sajip]

Vinay Sajip  added the comment:

> But I understand if this is out of scope.

I think it's out of scope here (as discussion of such issues will obscure the 
basic point about how to use the namer and rotator).

However, it might be useful to have a "gotchas" section somewhere in the 
logging docs called something like "Things to be careful about", 
"Anti-patterns", "Patterns to Avoid" or similar. Or maybe have it in the 
cookbook with a section at the end - "Recipes to Avoid".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[mori-b]

mori-b  added the comment:

Indeed this situation would rise only under misusage of the log, where multiple 
processes inherit by  mistake or by wrong design a file descriptor of a same 
log file (Regarding the threads case I unfortunately cannot reproduce).
While this kind of misusage doesn't have terrible consequences when not using 
the rotator, the growing used disk space which occurs when using the rotator is 
more serious, that's why I found appropriate to try and protect the programmer 
from his own mistake. But I understand if this is out of scope.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[Vinay Sajip]

Vinay Sajip  added the comment:

The cookbook example is (by design) limited in scope and doesn't especially 
discuss usage in multi-thread or multi-process scenarios - as with other 
examples in the cookbook which aren't specifically concerned with such 
scenarios. I don't think this is a problem with the example.

The problem occurs because on POSIX-style platforms, you can have multiple 
sources pointing to a file, and if deleted it only actually disappears when 
there are no references to it (such as open file descriptors/handles in other 
threads/processes). On Windows, for example, you would typically get a "sharing 
violation" type of error because you can't have multiple concurrent writers to 
a given file. This is not specifically a logging issue, as you could get the 
same thing happening with other kinds of files shared between processes and/or 
threads.

Adding os.truncate() might just serve to hide the problem for a while, but the 
right answer is to avoid opening the same log file multiple times. In the 
examples you mention, do multiple threads create handlers pointing to the same 
file? That might be an anti-pattern. Normally, the main thread configures 
logging (early in the "if __name__ == '__main__'" clause which creates the 
relevant handlers) and then any subsequently created threads just log to their 
loggers, but don't create any new handlers in the thread.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[Raymond Hettinger]

Change by Raymond Hettinger :


--
assignee: docs@python -> vinay.sajip
nosy: +vinay.sajip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[mori-b]

mori-b  added the comment:

Additional precision : this issue can happen when the log file is shared 
between multiple threads. And naturally also between different processes, which 
is not recommended but can happen by mistake.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[mori-b]

mori-b  added the comment:

In 
https://docs.python.org/3/howto/logging-cookbook.html#using-a-rotator-and-namer-to-customize-log-rotation-processing,
 the log rotator example deletes the original log file after compressing it. 
However, running on Linux the command "lsof +L1" shows that the deleted 
original log file might still hold the same disk space, and keep growing.
Replacing the command "os.remove(source)" with "os.truncate(source,0)" seems to 
solve the issue by freeing the original log file disk space at each rotation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44399] log rotator cookbook example might waste disk space

[mori-b]

New submission from mori-b :

In 
https://docs.python.org/3/howto/logging-cookbook.html#using-a-rotator-and-namer-to-customize-log-rotation-processing,
 the log rotator example deletes the original log file after compressing it. 
However, running on Linux the command "lsof +S1" shows that the deleted 
original log file might still hold the same disk space, and keep growing.
Replacing the command "os.remove(source)" with "os.truncate(source,0)" seems to 
solve the issue by freeing the original log file disk space at each rotation.

--
assignee: docs@python
components: Documentation
messages: 395658
nosy: docs@python, mori-b
priority: normal
severity: normal
status: open
title: log rotator cookbook example might waste disk space
type: resource usage

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com