This is a quick synopsis of my environment:
Ubuntu 12.04 64-bit Linux
Python 2.7.3
PyInstaller 2.1
I have a *very* simple Python program (test2.py) that merely prints
the file system encoding as follows:
import sys
print "Getfilesystemencoding: ", sys.getfilesystemencoding()
If run from the command-line (unfrozen) I get this output:
$python test2.py
Getfilesystemencoding: UTF-8
If I "freeze" it via PyInstaller (either --onedir or --onefile) I get
the following output:
$pyinstaller --onefile test2.py
6 INFO: wrote /home/gschmottlach/Junk/pytest/test2.spec
25 INFO: UPX is not available.
43 INFO: Processing hook hook-os
111 INFO: Processing hook hook-time
135 INFO: Processing hook hook-_sre
150 INFO: Processing hook hook-cStringIO
156 INFO: Processing hook hook-codecs
163 INFO: Processing hook hook-encodings
478 INFO: Processing hook hook-cPickle
549 INFO: Extending PYTHONPATH with /home/gschmottlach/Junk/pytest
550 INFO: checking Analysis
550 INFO: building Analysis because out00-Analysis.toc non existent
550 INFO: running Analysis out00-Analysis.toc
648 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/_pyi_bootstrap.py
665 INFO: Processing hook hook-os
675 INFO: Processing hook hook-site
685 INFO: Processing hook hook-encodings
750 INFO: Processing hook hook-time
774 INFO: Processing hook hook-_sre
790 INFO: Processing hook hook-cStringIO
796 INFO: Processing hook hook-codecs
1118 INFO: Processing hook hook-cPickle
1220 INFO: Processing hook hook-pydoc
1299 INFO: Processing hook hook-email
1337 INFO: Processing hook hook-httplib
1361 INFO: Processing hook hook-email.message
1398 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_importers.py
1432 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_archive.py
1465 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_carchive.py
1497 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_os_path.py
1501 INFO: Analyzing test2.py
1501 INFO: Hidden import 'codecs' has been found otherwise
1501 INFO: Hidden import 'encodings' has been found otherwise
1501 INFO: Looking for run-time hooks
objdump: section '.dynamic' mentioned in a -j option, but not found in
any input file
1886 INFO: Using Python library /usr/lib/libpython2.7.so.1.0
1886 INFO: Adding Python library to binary dependencies
1903 INFO: Warnings written to
/home/gschmottlach/Junk/pytest/build/test2/warntest2.txt
1908 INFO: checking PYZ
1908 INFO: rebuilding out00-PYZ.toc because out00-PYZ.pyz is missing
1908 INFO: building PYZ (ZlibArchive) out00-PYZ.toc
2496 INFO: checking PKG
2496 INFO: rebuilding out00-PKG.toc because out00-PKG.pkg is missing
2496 INFO: building PKG (CArchive) out00-PKG.pkg
objdump: section '.dynamic' mentioned in a -j option, but not found in
any input file
4112 INFO: checking EXE
4112 INFO: rebuilding out00-EXE.toc because test2 missing
4112 INFO: building EXE from out00-EXE.toc
4113 INFO: Appending archive to EXE /home/gschmottlach/Junk/pytest/dist/test2
And then run it:
$ ./dist/test2
Getfilesystemencoding: None
I've compared environments, LC_* variables etc.. and cannot see any
appreciable difference. I also created another more complicated test
program that provides additional diagnostics:
import sys
import os
import codecs
from ctypes import *
import locale
CODESET = 14
LC_CTYPE = 0
locale.setlocale(locale.LC_ALL, '')
c = codecs.lookup('utf-8')
libc = CDLL("libc.so.6")
nl_langinfo = libc.nl_langinfo
nl_langinfo.argtypes = [c_int]
nl_langinfo.restype = c_char_p
setlocale = libc.setlocale
setlocale.argtypes = [c_int, c_char_p]
setlocale.restype = c_char_p
print "nl_langinfo(CODESET): ", libc.nl_langinfo(14)
print "Getfilesystemencoding: ", sys.getfilesystemencoding()
print "LANG: ", os.getenv('LANG')
print "Codecs: ", c.name
print "Current locale: ", setlocale(LC_CTYPE, None)
setlocale(LC_CTYPE, "")
encoding = nl_langinfo(CODESET)
print "nl_langinfo(CODESET): ", encoding
try:
encoder = codecs.getencoder(encoding)
print "encoder[%s]: %r" % (encoding, encoder)
except LookupError, e:
print "Failed to lookup %s : %r" % (encoding, e)
Running this from the command-line (unfrozen) I get the following output:
$ python test.py
nl_langinfo(CODESET): UTF-8
Getfilesystemencoding: UTF-8
LANG: en_US.UTF-8
Codecs: utf-8
Current locale: en_US.UTF-8
nl_langinfo(CODESET): UTF-8
encoder[UTF-8]: <built-in function utf_8_encode>
$ pyinstaller --onefile test.py
6 INFO: wrote /home/gschmottlach/Junk/pytest/test.spec
25 INFO: UPX is not available.
42 INFO: Processing hook hook-os
111 INFO: Processing hook hook-time
135 INFO: Processing hook hook-_sre
151 INFO: Processing hook hook-cStringIO
157 INFO: Processing hook hook-codecs
164 INFO: Processing hook hook-encodings
477 INFO: Processing hook hook-cPickle
551 INFO: Extending PYTHONPATH with /home/gschmottlach/Junk/pytest
552 INFO: checking Analysis
552 INFO: building Analysis because out00-Analysis.toc non existent
552 INFO: running Analysis out00-Analysis.toc
642 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/_pyi_bootstrap.py
660 INFO: Processing hook hook-os
671 INFO: Processing hook hook-site
682 INFO: Processing hook hook-encodings
748 INFO: Processing hook hook-time
773 INFO: Processing hook hook-_sre
789 INFO: Processing hook hook-cStringIO
795 INFO: Processing hook hook-codecs
1121 INFO: Processing hook hook-cPickle
1223 INFO: Processing hook hook-pydoc
1302 INFO: Processing hook hook-email
1340 INFO: Processing hook hook-httplib
1364 INFO: Processing hook hook-email.message
1403 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_importers.py
1437 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_archive.py
1470 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_carchive.py
1503 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_os_path.py
1507 INFO: Analyzing test.py
1615 INFO: Hidden import 'codecs' has been found otherwise
1616 INFO: Hidden impo6 INFO: wrote /home/gschmottlach/Junk/pytest/test.spec
25 INFO: UPX is not available.
42 INFO: Processing hook hook-os
111 INFO: Processing hook hook-time
135 INFO: Processing hook hook-_sre
151 INFO: Processing hook hook-cStringIO
157 INFO: Processing hook hook-codecs
164 INFO: Processing hook hook-encodings
477 INFO: Processing hook hook-cPickle
551 INFO: Extending PYTHONPATH with /home/gschmottlach/Junk/pytest
552 INFO: checking Analysis
552 INFO: building Analysis because out00-Analysis.toc non existent
552 INFO: running Analysis out00-Analysis.toc
642 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/_pyi_bootstrap.py
660 INFO: Processing hook hook-os
671 INFO: Processing hook hook-site
682 INFO: Processing hook hook-encodings
748 INFO: Processing hook hook-time
773 INFO: Processing hook hook-_sre
789 INFO: Processing hook hook-cStringIO
795 INFO: Processing hook hook-codecs
1121 INFO: Processing hook hook-cPickle
1223 INFO: Processing hook hook-pydoc
1302 INFO: Processing hook hook-email
1340 INFO: Processing hook hook-httplib
1364 INFO: Processing hook hook-email.message
1403 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_importers.py
1437 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_archive.py
1470 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_carchive.py
1503 INFO: Analyzing
/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1-py2.7.egg/PyInstaller/loader/pyi_os_path.py
1507 INFO: Analyzing test.py
1615 INFO: Hidden import 'codecs' has been found otherwise
1616 INFO: Hidden import 'encodings' has been found otherwise
1616 INFO: Looking for run-time hooks
objdump: section '.dynamic' mentioned in a -j option, but not found in
any input file
2043 INFO: Using Python library /usr/lib/libpython2.7.so.1.0
2044 INFO: Adding Python library to binary dependencies
2068 INFO: Warnings written to
/home/gschmottlach/Junk/pytest/build/test/warntest.txt
2074 INFO: checking PYZ
2074 INFO: rebuilding out00-PYZ.toc because out00-PYZ.pyz is missing
2074 INFO: building PYZ (ZlibArchive) out00-PYZ.toc
2709 INFO: checking PKG
2709 INFO: rebuilding out00-PKG.toc because out00-PKG.pkg is missing
2709 INFO: building PKG (CArchive) out00-PKG.pkg
objdump: section '.dynamic' mentioned in a -j option, but not found in
any input file
4647 INFO: checking EXE
4647 INFO: rebuilding out00-EXE.toc because test missing
4647 INFO: building EXE from out00-EXE.toc
4647 INFO: Appending archive to EXE
/home/gschmottlach/Junk/pytest/dist/testrt 'encodings' has been found
otherwise
1616 INFO: Looking for run-time hooks
objdump: section '.dynamic' mentioned in a -j option, but not found in
any input file
2043 INFO: Using Python library /usr/lib/libpython2.7.so.1.0
2044 INFO: Adding Python library to binary dependencies
2068 INFO: Warnings written to
/home/gschmottlach/Junk/pytest/build/test/warntest.txt
2074 INFO: checking PYZ
2074 INFO: rebuilding out00-PYZ.toc because out00-PYZ.pyz is missing
2074 INFO: building PYZ (ZlibArchive) out00-PYZ.toc
2709 INFO: checking PKG
2709 INFO: rebuilding out00-PKG.toc because out00-PKG.pkg is missing
2709 INFO: building PKG (CArchive) out00-PKG.pkg
objdump: section '.dynamic' mentioned in a -j option, but not found in
any input file
4647 INFO: checking EXE
4647 INFO: rebuilding out00-EXE.toc because test missing
4647 INFO: building EXE from out00-EXE.toc
4647 INFO: Appending archive to EXE /home/gschmottlach/Junk/pytest/dist/test
Running the frozen application I see:
$ ./dist/test
nl_langinfo(CODESET): UTF-8
Getfilesystemencoding: None
LANG: en_US.UTF-8
Codecs: utf-8
Current locale: en_US.UTF-8
nl_langinfo(CODESET): UTF-8
encoder[UTF-8]: <built-in function utf_8_encode>
Looking at the Python source code, specifically pythonrun.c in the
function Py_InitializeEx() it looks like Py_FileSystemDefaultEncoding,
a "const char *", is initialized there after initally be set to NULL
in bltinmodule.c@25. This variable is returned by the call to
sys.getfilesystemencoding() from Python. I've followed the code path
and the only thing I can think of is that there is a call to
PyCodec_Encoder(loc_codeset) that may be made *before* PyInstaller has
had a chance to install it's own module loading hooks (e.g. to
re-direct where the codec module it loaded from). So, my suspicion is
that there is a race-condition here where PyInstaller cannot re-direct
the search into it's internal TOC of archived/frozen modules.
Has anyone else seen this behavior? Does anyone with more
understanding of PyInstaller have a suggestion for a fix?
Thanks for any insights that can be offered. It seems quite similar to
bug #885 reported in the old TRAC bug-tracker.
--
You received this message because you are subscribed to the Google Groups
"PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/pyinstaller.
For more options, visit https://groups.google.com/d/optout.
