Re: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version
Hi,
First of all, I just found an old issue that we will solved by my PEP 587 :-)
Add Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal errors
https://bugs.python.org/issue30560
I studied code of applications embedding Python. Most of them has to
decode bytes strings to get wchar_t* to set home, argv, program name,
etc. I'm not sure that they use the "correct" encoding, especially
since Python 3.7 got UTF-8 Mode (PEP 540) and C locale coercion (PEP
538).
I tried to convert the source code of each project into pseudo-code
which looks like C code used in CPython.
I removed all error handling code: look at each reference, the
original code is usually way more complex.
Some project has to wrap each function of the Python C API manually,
which adds even more boilerplate code.
Some project set/unset environment varaibles. Others prefer global
configuration variables like Py_NoSiteFlag.
It seems like Py_FrozenFlag is commonly used. Maybe I should make the
flag public and try to find it a better name:
/* If greater than 0, suppress _PyPathConfig_Calculate() warnings.
If set to -1 (default), inherit Py_FrozenFlag value. */
int _frozen;
About pyinstaller which changes C standard stream buffering:
Py_Initialize() now also does that when buffered_stdio=0. See
config_init_stdio() in Python/coreconfig.c. Moreover, this function
now *always* set standard streams to O_BINARY mode on Windows. I'm not
sure if it's correct or not.
Blender
---
Pseudo-code of BPY_python_start::
BLI_strncpy_wchar_from_utf8(program_path_wchar, BKE_appdir_program_path());
Py_SetProgramName(program_path_wchar);
PyImport_ExtendInittab(bpy_internal_modules);
Py_SetPythonHome(py_path_bundle_wchar);
Py_SetStandardStreamEncoding("utf-8", "surrogateescape");
Py_NoSiteFlag = 1;
Py_FrozenFlag = 1;
Py_Initialize();
Ref:
https://git.blender.org/gitweb/gitweb.cgi/blender.git/blob/HEAD:/source/blender/python/intern/bpy_interface.c
fontforge
-
Pseudo-code of fontforge when Python is used to run a script::
Py_Initialize()
for init_file in init_files:
PyRun_SimpleFileEx(init_file)
exitcode = Py_Main(arg, argv)
Py_Finalize()
exit(exitcode)
Ref: https://bugs.python.org/issue36204#msg337256
py2app
--
Pseudo-code::
unsetenv("PYTHONOPTIMIZE");
unsetenv("PYTHONDEBUG");
unsetenv("PYTHONDONTWRITEBYTECODE");
unsetenv("PYTHONIOENCODING");
unsetenv("PYTHONDUMPREFS");
unsetenv("PYTHONMALLOCSTATS");
setenv("PYTHONDONTWRITEBYTECODE", "1", 1);
setenv("PYTHONUNBUFFERED", "1", 1);
setenv("PYTHONPATH", build_python_path(), 1);
setlocale(LC_ALL, "en_US.UTF-8");
mbstowcs(w_program, c_program, PATH_MAX+1);
Py_SetProgramName(w_program);
Py_Initialize()
argv_new[0] = _Py_DecodeUTF8_surrogateescape(script, strlen(script));
...
PySys_SetArgv(argc, argv_new);
PyRun_SimpleFile(fp, script);
Py_Finalize();
Ref:
https://bitbucket.org/ronaldoussoren/py2app/src/default/py2app/apptemplate/src/main.c
See also:
https://bitbucket.org/ronaldoussoren/py2app/src/default/py2app/bundletemplate/src/main.m
OpenOffice
--
Pseudo-code of ``PythonInit``::
mbstowcs(wide, home, PATH_MAX + 1);
Py_SetPythonHome(wide);
setenv("PYTHONPATH", getenv("PYTHONPATH") + ":" + path_bootstrap);
PyImport_AppendInittab("pyuno", PyInit_pyuno);
Py_DontWriteBytecodeFlag = 1;
Py_Initialize();
Ref: pyuno/source/loader/pyuno_loader.cxx, see:
https://docs.libreoffice.org/pyuno/html/pyuno__loader_8cxx_source.html
vim
---
Pseudo-code::
mbstowcs(py_home_buf, p_py3home);
Py_SetPythonHome(py_home_buf);
PyImport_AppendInittab("vim", Py3Init_vim);
Py_Initialize();
Ref: https://github.com/vim/vim/blob/master/src/if_python3.c
pyinstaller
---
Pseudo-code::
pyi_locale_char2wchar(progname_w, status->archivename)
SetProgramName(progname_w);
pyi_locale_char2wchar(pyhome_w, status->mainpath)
SetPythonHome(pyhome_w);
pypath_w = build_path();
Py_SetPath(pypath_w);
Py_NoSiteFlag = 1;
Py_FrozenFlag = 1;
Py_DontWriteBytecodeFlag = 1;
Py_NoUserSiteDirectory = 1;
Py_IgnoreEnvironmentFlag = 1;
Py_VerboseFlag = 0;
Py_OptimizeFlag = 1;
if (unbuffered) {
#ifdef _WIN32
_setmode(fileno(stdin), _O_BINARY);
_setmode(fileno(stdout), _O_BINARY);
#endif
setbuf(stdin, (char *)NULL);
setbuf(stdout, (char *)NULL);
setbuf(stderr, (char *)NULL);
}
Py_Initialize();
PySys_SetPath(pypath_w);
PySys_SetArgvEx(argc, wargv, 0);
Ref:
https://github.com/pyinstaller/pyinstaller/blob/1844d69f5aa1d64d3feca912ed1698664a3faf3e/bootloader/src/pyi_pythonlib.c
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
Re: [Python-Dev] Ignore spurious 3.4 checkin messages!
On Fri, 10 May 2019 at 18:54, Ned Deily wrote: > > Eek! I was just doing a bit of branch cleanup in the cpython repo and > managed to trigger a bunch (30-ish) of duplicate checkin messages to > bugs.python.org for old commits. I will remove them from b.p.o. Please > ignore any 3.4 spam email. Sorry! It's not dead! See, it just moved! ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2019-05-03 - 2019-05-10) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open7084 ( +3) closed 41542 (+88) total 48626 (+91) Open issues with patches: 2851 Issues opened (65) == #15987: Provide a way to compare AST nodes for equality recursively https://bugs.python.org/issue15987 reopened by vstinner #36785: Implement PEP 574 https://bugs.python.org/issue36785 opened by pitrou #36786: "make install" should run compileall in parallel https://bugs.python.org/issue36786 opened by pitrou #36788: Add clamp() function to builtins https://bugs.python.org/issue36788 opened by TheComet #36789: Unicode HOWTO incorrectly states that UTF-8 contains no zero b https://bugs.python.org/issue36789 opened by mbiggs #36790: test_asyncio fails with application verifier! https://bugs.python.org/issue36790 opened by Alexander Riccio #36791: sum() relies on C signed overflow behaviour https://bugs.python.org/issue36791 opened by serhiy.storchaka #36792: [Windows] time: crash on formatting time with de_DE locale https://bugs.python.org/issue36792 opened by CharlieClark #36793: Do not define unneeded __str__ equal to __repr__ https://bugs.python.org/issue36793 opened by serhiy.storchaka #36794: asyncio.Lock documentation in Py3.8 lacks parts presented in d https://bugs.python.org/issue36794 opened by germn #36795: "make venv" failed in Docs https://bugs.python.org/issue36795 opened by pitrou #36796: Error handling cleanup in _testcapimodule.c https://bugs.python.org/issue36796 opened by ZackerySpytz #36797: Cull more oudated distutils information https://bugs.python.org/issue36797 opened by ncoghlan #36799: Typo in ctypes documentation https://bugs.python.org/issue36799 opened by rrt #36803: Getting a lot of runtime misaligned address error while buildi https://bugs.python.org/issue36803 opened by dibya ranjan mishra #36804: Python for Windows installer Repair option does not repair PIP https://bugs.python.org/issue36804 opened by Mark Green #36805: Don't close subprocess stream if it's stdin is closed https://bugs.python.org/issue36805 opened by asvetlov #36807: IDLE doesn't call os.fsync() https://bugs.python.org/issue36807 opened by gvanrossum #36809: Crash for test test_importlib https://bugs.python.org/issue36809 opened by furzoom #36810: Recursive type annotations do not work in documentation tests https://bugs.python.org/issue36810 opened by lovasoa #36813: QueueListener not calling task_done upon termination https://bugs.python.org/issue36813 opened by bar.harel #36816: self-signed.pythontest.net TLS certificate key is too weak https://bugs.python.org/issue36816 opened by gregory.p.smith #36818: Add PyInterpreterState.runtime. https://bugs.python.org/issue36818 opened by eric.snow #36819: Crash during encoding using UTF-16/32 and custom error handler https://bugs.python.org/issue36819 opened by atalaba #36820: Captured exceptions are keeping user objects alive unnecessari https://bugs.python.org/issue36820 opened by mariocj89 #36821: Termios module largely untested https://bugs.python.org/issue36821 opened by anthonypjshaw #36822: Minor grammatical fix in glossary.rst https://bugs.python.org/issue36822 opened by CuriousLearner #36823: shutil.copytree copies directories and files but fails with th https://bugs.python.org/issue36823 opened by rbavery #36824: Refactor str tests to reflect that str and unicode are merged https://bugs.python.org/issue36824 opened by dfortunov #36825: Make TestCase aware of the command line arguments given to Tes https://bugs.python.org/issue36825 opened by remi.lapeyre #36826: ast_unparser.c doesn't handle := expressions https://bugs.python.org/issue36826 opened by eric.smith #36827: Overriding __new__ method with itself changes behaviour of the https://bugs.python.org/issue36827 opened by alexey-muranov #36829: CLI option to make PyErr_WriteUnraisable abort the current pro https://bugs.python.org/issue36829 opened by graingert #36833: Add tests for Datetime C API Macros https://bugs.python.org/issue36833 opened by p-ganssle #36834: mock.patch.object does not persist __module__ name for functio https://bugs.python.org/issue36834 opened by spark #36837: Make il8n tools available from `python -m` https://bugs.python.org/issue36837 opened by bbkane #36839: Support the buffer protocol in code objects https://bugs.python.org/issue36839 opened by dino.viehland #36840: Add stream.abort() async method https://bugs.python.org/issue36840 opened by asvetlov #36841: Supporting customization of float encoding in JSON https://bugs.python.org/issue36841 opened by mitar #36842: Implement PEP 578 https://bugs.python.org/issue36842 opened by steve.dower #36843: AIX build fails with failure to get random numbers https://bugs.python.org/issue36843 opened
[Python-Dev] Ignore spurious 3.4 checkin messages!
Eek! I was just doing a bit of branch cleanup in the cpython repo and managed to trigger a bunch (30-ish) of duplicate checkin messages to bugs.python.org for old commits. I will remove them from b.p.o. Please ignore any 3.4 spam email. Sorry! -- Ned Deily n...@python.org -- [] ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 580/590 discussion
On 2019-05-10 00:07, Petr Viktorin wrote: METH_FASTCALL is currently not documented, and it should be renamed before it's documented. Names with "fast" or "new" generally don't age well. Just to make sure that we're understanding correctly, is your proposal to do the following: - remove the name METH_FASTCALL - remove the calling convention METH_FASTCALL without METH_KEYWORDS - rename METH_FASTCALL|METH_KEYWORDS -> METH_VECTORCALL ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 580/590 discussion
Petr Viktorin schrieb am 10.05.19 um 00:07: > On 5/9/19 5:33 PM, Jeroen Demeyer wrote: >> Maybe you misunderstood my proposal. I want to allow both for extra >> flexibility: >> >> - METH_FASTCALL (possibly combined with METH_KEYWORDS) continues to work >> as before. If you don't want to care about the implementation details of >> vectorcall, this is the right thing to use. >> >> - METH_VECTORCALL (using exactly the vectorcallfunc signature) is a new >> calling convention for applications that want the lowest possible >> overhead at the cost of being slightly harder to use. > > Then we can, in the spirit of minimalism, not add METH_VECTORCALL at all. > [...] > METH_FASTCALL is currently not documented, and it should be renamed before > it's documented. Names with "fast" or "new" generally don't age well. I personally don't see an advantage in having both, apart from helping code that wants to be fast also on Py3.7, for example. It unnecessarily complicates the CPython implementation and C-API. I'd be ok with removing FASTCALL in favour of VECTORCALL. That's more code to generate for Cython in order to adapt to Py<3.6, Py3.6, Py3.7 and then Py>=3.[89], but well, seeing the heap of code that we *already* generate, it's not going to hurt our users much. It would, however, be (selfishly) helpful if FASTCALL could still go through a deprecation period, because we'd like to keep the current Cython 0.29.x release series compatible with Python 3.8, and I'd like to avoid adding support for VECTORCALL and compiling out FASTCALL in a point release. Removing it in Py3.9 seems ok to me. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a tzidx cache to datetime
On 10.05.2019 02:58, Paul Ganssle wrote: > This is only "only" for dateutil in the sense that no one other than > dateutil implements tzinfo with the interface provided. If dateutil were > /not/ already implemented with a list of offsets and their indexes, I > would still propose this, and just re-write dateutil to take advantage > of it. From a cursory glance at pendulum, it seems that they could take > advantage of it as well (though they use their own datetime subclass, so > they have always had the ability to add this). > >> What do you think of adding a private "_cache" attribute which would >> be an arbitrary Python object? (None by default) > > We cannot use a private attribute (other than to do the actual storage, > since the thing that gets stored is not directly accessible anyway and > is instead mediated by a layer that manages the cache) because this is a > feature explicitly being added for use by tzinfo, /not/ by datetime. If > it's private then it's not safe for implementations of tzinfo to > actually use it, which defeats the purpose. > > Regarding the use of an arbitrary Python object: What I'm proposing is > that we offer a bit of the "free" storage space in the alignment bits to > tzinfo objects to use as a cache. In /most/ cases this will be very > useful to someone implementing a tzinfo, because there are really only > so many ways to accomplish this task, and most time zones are > expressible as a very short list of offset/name/dst combinations, plus > some rule for which applies when, which is why a small integer cache is > sufficient and more or less universal (i.e. not specific to dateutil's > implementation). > > I will also note that in my design, it is still possible for `tzinfo` to > return something other than [0, 254], it's just that that information > will not be cached, so it won't get the benefit of any optimization, but > the same interface / implementation can be used. > > In my test with gcc, adding an additional PyObject* to the end of the > PyDateTime_DateTime struct increased the size of the `datetime.datetime` > object from 64 to 72 bytes, whereas adding an `unsigned char` after the > `fold` leaves it unchanged. Given that the expansion to arbitrary Python > objects is speculative and doesn't have any particular use case, I would > prefer to leave the feature as is, and reconsider the possibility of > storing arbitrary Python objects on the datetime if there's some > compelling reason to do so (it would be a backwards-compatible change at > that point anyway). Given that many datetime objects in practice don't use timezones (e.g. in large data stores you typically use UTC and naive datetime objects), I think that making the object itself larger to accommodate for a cache, which will only be used a smaller percentage of the use cases, isn't warranted. Going from 64 bytes to 72 bytes also sounds like this could have negative effects on cache lines. If you need a per object cache, you can either use weakref objects or maintain a separate dictionary in dateutil or other timezone helpers which indexes objects by id(obj). That said, if you only add a byte field which doesn't make the object larger in practice (you merely use space that alignments would use anyway), this shouldn't be a problem. The use of that field should be documented, though, so that other implementations can use/provide it as well. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 10 2019) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ > On 5/9/19 8:14 PM, Victor Stinner wrote: >> Hi Paul, >> >> The change is basically an optimization. I'm uncomfortable to design >> it "only" for dateutil. What if tomorrow someone has to store an >> arbitrary Python object, rather than just an integer (in range [0; >> 254]), into a datetime for a different optimization? >> >> Moreover, I dislike adding a *public* method for an *internal* cache. >> >> Right now, it is not possible to create a weak reference to a >> datetime. If we make it possible, it would be possible to have an >> external cache implemented with weakref.WeakSet to clear old entries >> when a datetime object is detroyed. >> >> What do you think of adding a private "_cache" attribute which would >> be an arbitrary Python object? (None by default) >> >> Victor >> >> Le mar. 7 mai
