[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Sye van der Veen]

Sye van der Veen added the comment:

I would also agree that failing to load the main codec should be an abort.  
This bug was specifically the race condition in writing the .pyc file.

Thanks for all your help!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Eric Snow]

Eric Snow added the comment:

With issue #28007 wrapped up, there isn't a lot left to do here.  I was 
considering that we don't want to abort when we have problems loading a codec 
during startup.  However, Steve Dower made the point to me that a problem with 
the main codec during startup should be fatal.  Consequently I'm closing this 
bug.

--
resolution:  -> duplicate
stage: patch review -> resolved
status: open -> closed
superseder:  -> Bad .pyc files prevent import of otherwise valid .py files.

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Eric Snow]

Eric Snow added the comment:

I've opened #28007 to cover the concerns about bad a .pyc file blocking import 
from a valid .py file.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Eric Snow]

Changes by Eric Snow :


Removed file: http://bugs.python.org/file44424/issue16384.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Eric Snow]

Eric Snow added the comment:

Here's an updated patch.  Per a suggestion from Brett, I've chained the 
original EOFError with an ImportError.  The consequence is that the problematic 
encoding is skipped (silently) rather than causing the interpreter to abort.

FYI, I've opened issue #28005 to address how we silently skip encodings that 
fail to import.

--
Added file: http://bugs.python.org/file6/issue16384.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Brett Cannon]

Brett Cannon added the comment:

It's a question of whether you want the error that your .pyc files have somehow 
ended up in a bad state to pass silently or you should know you that something 
on your system is corrupting .pyc files (hence why the EOFError has been 
allowed to propagate).

Making corrupt .pyc files pass silently would be a change in semantics and 
practice, so we need to think through the ramifications. At worst we would need 
to raise a warning that such a thing happened, if not keep the current 
semantics.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Sye van der Veen]

Sye van der Veen added the comment:

I feel this patch (file44424) misses the mark. Any two Python processes
that try to import a module, without a pyc, at the same time could suffer
race conditions.  The first process will start to write the pyc, get
interrupted, and the second will fail with an EOFError.

When importing encodings at startup, this is a nasty abort. But it's just
as nasty if a running script gets unpredictable EOFErrors.

Corrupt .pyc files should not prevent importing if the valid .py files are
available.

On Tue, Sep 6, 2016, 11:29 PM Eric Snow  wrote:

>
> Eric Snow added the comment:
>
> After looking more closely, it looks like we should be ignoring such bogus
> modules.  Here's a patch to do so.
>
> --
> keywords: +patch
> stage: test needed -> patch review
> Added file: http://bugs.python.org/file44424/issue16384.diff
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Eric Snow]

Eric Snow added the comment:

After looking more closely, it looks like we should be ignoring such bogus 
modules.  Here's a patch to do so.

--
keywords: +patch
stage: test needed -> patch review
Added file: http://bugs.python.org/file44424/issue16384.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Eric Snow]

Eric Snow added the comment:

I have verified that this is still a problem (basically 3.6b1).

Fatal Python error: Py_Initialize: Unable to get the locale encoding
Traceback (most recent call last):
  File "/opt/python3.6/lib/python3.6/encodings/__init__.py", line 99, in 
search_function
  File "", line 979, in _find_and_load
  File "", line 968, in _find_and_load_unlocked
  File "", line 673, in _load_unlocked
  File "", line 663, in exec_module
  File "", line 768, in get_code
  File "", line 478, in _compile_bytecode
EOFError: marshal data too short
Aborted (core dumped)

We should be exiting with an error rather than aborting.

--
components: +Interpreter Core -None
stage:  -> test needed
versions: +Python 3.6 -Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Brett Cannon]

Changes by Brett Cannon br...@python.org:


--
nosy: +brett.cannon

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Brett Cannon]

Changes by Brett Cannon br...@python.org:


--
type: crash - behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Terry J. Reedy]

Terry J. Reedy added the comment:

The import machinery was changed in 3.3.0. I suggest checking if it gives you 
*exactly* the same behavior.

--
nosy: +terry.reedy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16384] import.c doesn't handle EOFError from PyMarshal_Read*

[Sye van der Veen]

New submission from Sye van der Veen:

The PyMarshal_Read* functions raise EOFError when the end of the file is 
unexpectedly met.  The current import.c functions propagate this error when 
reading .pyc or .pyo files.  One consequence of this is that Python will abort 
on startup if, say, encodings.utf_8's .pyc file is truncated.

I have encountered a scenario where this truncation is common.  If a second 
Python process is launched while the first is writing the core .pyc files, 
the second process may open the files before they are completely written and 
will thus see truncated files and abort.  This is a race condition that I was 
able to reproduce consistently on several Windows Server 2008 RC2 Standard SP1 
machines running 32-bit Python 3.2.3 from GNU make with -j 16 (Intel Xeon 
E5405 2GHz 2 processors 8GB 64-bit OS).  (Of course, I had to clean the 
__pycache__ directories between tests.)

This can be fixed in load_source_module by making read_compiled_module failures 
non-fatal:
if (cpathname != NULL 
(fpc = check_compiled_module(pathname, st.st_mtime, cpathname))) {
co = read_compiled_module(cpathname, fpc);
if (co == NULL) PyErr_Clear();
fclose(fpc);
}
if (co != NULL) {
// etc...
}
else {
co = parse_source_module(pathname, fp);
// etc...
write_compiled_module(co, cpathname, st);
}
This is similar to how write_compiled_module ignores failures in writing the 
.pyc files.  It ensures that if the .pyc file is corrupt for _any_ reason, it 
will get rewritten; this could be made specific to EOFError, but I don't 
recommed that.  Mostly, it ensures that corrupt .pyc files do not prevent 
Python from loading if the .py files are valid.

--
components: None
messages: 174438
nosy: syeberman
priority: normal
severity: normal
status: open
title: import.c doesn't handle EOFError from PyMarshal_Read*
type: crash
versions: Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com