[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[Inada Naoki]

Inada Naoki  added the comment:

The another error I found is already reported as #42868.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[Inada Naoki]

Change by Inada Naoki :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed
versions: +Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[Inada Naoki]

Inada Naoki  added the comment:

I confirmed that this bug is fixed, but I found another error.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[Inada Naoki]

Inada Naoki  added the comment:

Is this bug fixed by #26730?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[Graham Coster]

Graham Coster  added the comment:

This may be a silly question, however, does SpooledTemporaryFile need to exist 
at all?

>From some testing on macOS, SpooledTemporaryFile appeared to never have a 
>performance advantage over OS file caching, but with max_size greater than 
>4GB, it was a significant disadvantage.   

I found that the macOS built-in file cache was increasing in size as I wrote 
bigger TemporaryFile files, up to some limit the OS had decided. So, it seems 
the OS is automatically doing the same job as SpooledTemporaryFile.  Once the 
OS decided to write to disk, there was no sudden hit to performance, it just 
slowed down.

However, when SpooledTemporaryFile rolled-over large max_size files, there was 
a temporary big hit to performance, which then became a consistent slow down 
the same as TemporaryFile.

A big issue came with very large SpooledTemporaryFile  max_sizes hogging RAM 
and causing the OS to start swapping all processes.  This caused a huge 
performance hit to my program and the system as a whole. Once my program did 
finish, it took the system considerable time to reclaim swap.

I’m guessing SpooledTemporaryFile may have benefits on light weight embedded 
OSes that have no, or poor, file caching.  However, tuning the max_size to work 
with embedded systems’ limited RAM could be tricky for developers and would be 
hardware dependent. So, perhaps leaving file caching to operating systems is 
actually a better, and safer, option than offering it in Python?

If there are no benefits to SpooledTemporaryFile, should it be deprecated? If 
so, as it is phasesd out, could it be patched to be a TemporaryFile wrapper, 
with no rollover functionality?

> On 25 Nov 2019, at 1:37 pm, Inada Naoki  wrote:
> 
> 
> Inada Naoki  added the comment:
> 
> SpooledTemporaryFile has very serious bug which causes data corruption 
> (#26730).  Please don't use it with text mode until it is fixed.
> 
> --
> 
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[Inada Naoki]

Inada Naoki  added the comment:

SpooledTemporaryFile has very serious bug which causes data corruption 
(#26730).  Please don't use it with text mode until it is fixed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[R. David Murray]

R. David Murray  added the comment:

The docs currently say "The returned object is a file-like object whose _file 
attribute is either an io.BytesIO or io.StringIO object (depending on whether 
binary or text mode was specified) or a true file object, depending on whether 
rollover() has been called."  The fact that taking an iterator gets you 
whatever the *current* _file object is is implied by that but not made 
explicit.  A doc update to make that explicit would probably be appropriate.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[Graham Coster]

New submission from Graham Coster :

Hi there,

As demonstrated in the attached script, I found that a StopIteration exception 
occurs once a SpooledTemporaryFile has been rolled over from memory to disk, 
even though seek(0) had move the file position to the top of the file.

I had previously been using a TemporaryFile, which wrote directly to disk, 
without any issues.  However, once I swapped to a SpooledTemporaryFile, the 
StopIteration exception started to appear.  

I'm guessing the problem relates to the csv reader having an Iterator that is 
no longer looking at the correct file after the SpooledTemporaryFile is 
rolled-over.  I found that instantiating a new csv reader post roll-over works 
around the problem.

I had expected that I could just swap from a TemporaryFile to a 
SpooledTemporaryFile with no change to behaviour. However, if this is not the 
case, perhaps the SpooledTemporaryFile documentation should explicitly state 
this.

Cheers, Graham.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[Graham Coster]

Change by Graham Coster :


--
components: Library (Lib)
files: spooled-temp-file-does-not-seek-correctly-after-being-rolled.py
nosy: Gary Fernie, James Hennessy, graham.coster, inada.naoki, martin.panter, 
nubirstein, r.david.murray, serhiy.storchaka, terry.reedy
priority: normal
severity: normal
status: open
title: SpooledTemporaryFile does not seek correctly after being rolled over
type: behavior
versions: Python 3.7
Added file: 
https://bugs.python.org/file48685/spooled-temp-file-does-not-seek-correctly-after-being-rolled.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com