[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Vinay Sajip]

Vinay Sajip added the comment:

> Doesn't the "rotator" attribute break atomicity?

Which rotator do you mean? The default rotator is None, which leads to 
os.rename being called. If you're referring to the example in the documentation 
(cookbook) - it was intended purely as an example, and the paragraph under the 
snippet does say it's only intended for illustrative purposes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Doesn't the "rotator" attribute break atomicity?  A careful rotator should 
first rename the source to the temporary file, process the data and save it to 
other temporary file, and then rename the result to the destination and remove 
the first temporary file.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Antoine Pitrou]

Antoine Pitrou added the comment:

> I've updated the type to enhancement (it seems like a grey area to me
> - it's a behavioural fix for a niche use case).
> 
> I suggested a patch rather than simply subclassing RotatingFileHandler
> since:
>   - The subclass would just have a copy of RotatingFileHandler's
> doRollover method with a one-line change.
>   - The proposed fix is functionally equivalent to the current code
> for all currently working uses.

You may just as well monkeypatch os.rename() to fallback on
shutil.move() if the filenames are on a /dev/shm filesystem (or you
could bug QNX to fix their broken filesystem...).

>From a code quality and readability standpoint, os.rename() conveys the
intended semantics clearly, while shutil.move() doesn't, so switching to
shutil.move() in the stdlib would be a regression. Also, doing this in
logging would open the door to doing the same thing in other modules.
Even a critical piece of infrastructure such as importlib relies on
os.rename() working properly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Phil Connell]

Phil Connell added the comment:

I've updated the type to enhancement (it seems like a grey area to me - it's a 
behavioural fix for a niche use case).

I suggested a patch rather than simply subclassing RotatingFileHandler since:
  - The subclass would just have a copy of RotatingFileHandler's doRollover 
method with a one-line change.
  - The proposed fix is functionally equivalent to the current code for all 
currently working uses.


Thanks for the offer of having an extension linked from your blog. I don't 
think it's appropriate for this case (since it's just a slightly modified copy 
of the stdlib code - I'll probably just keep the patch along with a few other 
compatibility hacks we have) although I may have something for you in future 
(we do have a subclass that compresses old logs).

--
type: behavior -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Vinay Sajip]

Vinay Sajip added the comment:

The current implementation was written with an expectation of working rename 
functionality in the stdlib. As such, while this issue might be categorised as 
being of type "enhancement", I don't see how you can categorise it as being of 
type "behaviour".

What's the problem with subclassing the relevant handlers to implement your own 
doRollover() method? You only need to do that once for all the projects in your 
QNX environment. Given that it's not a mainstream environment, that's not IMO 
an unreasonable suggestion.

If you publish such a handler, I'll mention it on

http://plumberjack.blogspot.co.uk/p/handlers-for-logging.html

which has a whole bunch of handlers written by people for specific requirements 
which don't warrant adding functionality directly in the stdlib.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Phil Connell]

Phil Connell added the comment:

Serhiy, there are also calls to os.rename in RotatingFileHandler.doRollover

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Phil Connell]

Phil Connell added the comment:

I'm not convinced that it matters whether the rename or move is atomic. Can 
anyone come up with a quick concrete example?


I see two scenarios:

1. The process crashes during a copy in shutils.move(). In this case, some logs 
will be duplicated across the files involved in the copy.

2. Other threads emit logs during the rollover. Given that the IO lock is 
acquired in Handler.handle() before calling emit(), this is fine.


While the first case isn't ideal, I don't think there can be any loss of logs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

What about the "rotator" attribute.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Vinay Sajip]

Vinay Sajip added the comment:

Thanks for the patch, but I'm closing this as 'wontfix', as per the points made 
by Antoine and David. If you need logging from an embedded system, please 
consider using one of the socket-based logging handlers, if that's feasible in 
the specific situation.

--
resolution:  -> wont fix
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[R. David Murray]

R. David Murray added the comment:

I agree with Antoine.  It seems to me that it is very important to the 
semantics of rollover that the rename be atomic, even if we ignore the issue of 
existing other readers.  If it were not atomic, you might end up with lost log 
messages.  So I don't think there is anything to do here: you just can't use 
rollover on a filesystem that doesn't support atomic rename.

I'll leave it Vinay to close the issue if he agrees with us.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Antoine Pitrou]

Antoine Pitrou added the comment:

I'm not sure why you would setup logging on a RAM filesystem (I assume we're 
talking about normal volatile RAM)?
But besides, the big point of using rename() is that it's fast, atomic and 
won't disrupt existing file handlers pointing to that file. If 
RotatingFileHandler may resort to copying by mistake, it may cause other issues 
down the line.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Phil Connell]

Changes by Phil Connell :


--
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16449] RotatingFileHandler rollover doesn't work on QNX shmem filesystems

[Phil Connell]

New submission from Phil Connell:

logging.handlers.RotatingFileHandler.doRollover fails on QNX /dev/shmem 
filesystems (seen on a 6.4.0-based system).

QNX RAM filesystems don't support rename() (see 
http://www.qnx.com/developers/docs/6.4.0/neutrino/sys_arch/fsys.html#DEVSHMEM - 
it's a 'big filesystem' feature).

So for example, rename("/dev/shmem/foo", "/dev/shmem/bar") fails with EXDEV.


This is easily fixed by using shutils.move rather than os.rename where 
appropriate, falling back to copying if a rename() fails. It's not sufficient 
to set the rotator attribute, since there are other os.rename calls in in 
doRollover.

--
components: Library (Lib)
files: rfh_rename_fix.patch
keywords: patch
messages: 175277
nosy: pconnell, vinay.sajip
priority: normal
severity: normal
status: open
title: RotatingFileHandler rollover doesn't work on QNX shmem filesystems
versions: Python 2.7, Python 3.3
Added file: http://bugs.python.org/file27941/rfh_rename_fix.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com