[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks
On Sun, May 9, 2021 at 9:13 AM Antoine Pitrou wrote: > On Sun, 09 May 2021 02:16:02 - > "Jim J. Jewett" wrote: > > Antoine Pitrou wrote: > > > On Sat, 8 May 2021 02:58:40 + > > > Neil Schemenauer nas-pyt...@arctrix.com wrote: > > > > > > It would be cool if we could mmap the pyc files and have the VM run > > > > code without an unmarshal step. > > > > What happens if another process mutates or truncates the file while > the > > > CPython VM is executing code from the mapped file? Crash? > > > > Why would this be any different than whatever happens now? > > What happens now is that the pyc file is transferred at once to memory > using regular IO. So the chance is really slim that you read invalid > data due to concurrent mutation. > concurrent mutation isn't even what I was talking about. We don't protect against that today as that isn't a concern. But POSIX semantics on the bulk of systems where this would ever matter do software updates by moving new files into place. Because that is an idempotent inode change. So the existing open file already in the process of being read is not changed. But as soon as you do a new open call on the pathname you get a different file than the last time that path was opened. This is not theoretical. I've seen production problems as a result (zipimport - https://bugs.python.org/issue19081) making the incorrect assumption that they can reopen a file that they've read once at a later point in time. So if we do open files later, we must code defensively and assume they might not contain what we thought. We already have this problem with source code lines displayed in tracebacks today as those are read on demand. But as that is debugging information only the wrong source lines being shown next to the filename + linenumber in a traceback is something people just learn to ignore in these situations. We have the data to prevent this, we just never have. https://bugs.python.org/issue44091 filed to track that. Given this context, M.-A. Lemburg's alternative idea could have some merit as it would synchronize our source skew behavior with our additional debugging information behavior. My initial reaction is that it's falling into the trap of bundling too into one place though. quoting M.-A. Lemburg: > Create a new file format which supports enhanced debugging. This > would include the source code in a indexed format, the AST and > mappings between byte code, AST node, lines and columns. > > Python would then only use and load this file when it needs > to print a traceback - much like it does today with the source > code. > > The advantage is that you can add even more useful information > for debugging while not making the default code distribution > format take more memory (both disk and RAM). Realistically: This is going to take more disk space in the common case because in addition to the py, pyc, pyc.opt-1, pyc.opt-2 that some distros apparently include all of today, there'd be a new pyc.debuginfo to go along side it. The only benefit is that it isn't resident in ram. And someone *could* choose to filter these out of their distro or container or whatever-the-heck-their-package-format-is. But I really doubt that'll be the default. Not having debugging information when a problem you're trying to hunt down and reproduce but only happens once in a blue moon is extraordinarily frustrating. Which is why people who value engineering time deploy with debugging info. There are environments where people intentionally do not deploy source code. But do want to get debugging data from tracebacks that they can then correlate to their sources later for analysis (they're tracking exactly which versions of pycs from which versions of sources were deployed). It'd be a shame to exclude column information for this scenario. -gps ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6E7UZ5SFUAADUJUQ6DKPJIGO6CCGCNFU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Release of a responsive python-docs-theme 2021.5
> I wasn’t able to find examples - is this what is up and running on > docs.python.org, or is that a future plan? I believe currently only the pre-release (3.10) documentation https://docs.python.org/3.10/ and in-development (3.11) documentation https://docs.python.org/3.11/ is using the theme. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XI5N2TB3ARAFLV4U2JMU7NBTEK4VZ3ZO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Release of a responsive python-docs-theme 2021.5
Thanks, Carol (and Olga)! This sounds great! I’m sure you can all google :), but here’s the link: https://github.com/python/python-docs-theme I wasn’t able to find examples - is this what is up and running on docs.python.org, or is that a future plan? thanks much, —titus p.s. Not sure what address I’m subscribed to on python-dev, so this might get blocked for posting there. Is ok. > On May 7, 2021, at 4:32 PM, Carol Willing > wrote: > > It’s with great pleasure that I announce that python-docs-theme has been > released to PyPI. > > Thanks to the hard work and patience of Olga Bulat, @obulat, Python’s doc > theme is now responsive. Many thanks to everyone who has contributed to this > release by filing issues, writing PRs, reviewing PRs, and testing the theme. > It was a great team effort. > > Here are the highlights from the CHANGELOG.rst: > > - Make the theme responsive (#46) Contributed by Olga Bulat. > - Use Python 3.8 for the Github Actions (#71) Contributed by Stéphane Wirtel. > - Use default pygments theme (#68) Contributed by Aaron Carlisle. > - Test Github action to validate the theme against docsbuild scripts. (#69) > Contributed by Julien Palard. > - Add the copy button to pycon3 highlighted code blocks. (#64) Contributed by > Julien Palard. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/75SE5KPBFEWXOGOAFEKM6FBHJQ3AORXK/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2TN4YG7WXI4X6VUMO2F3KCNSWBG33AOE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks
On Sat, 8 May 2021 21:59:49 +0100 Pablo Galindo Salgado wrote: > > That could work, but in my personal opinion, I would prefer not to do > that as it complicates things and I think is overkill. > > Let me expand on this: > > I recognize the problem that -OO can be quite unusable if some of your > dependencies depend on docstrings and that It would be good to separate > this from that option, but I am afraid of the following: > > - New APIs in the marshal module and other places to pass down the extra > information to read/write or not the extra information. > - Complication of the pyc format with more entries in the header. > - Complication of the implementation. > > Given that the reasons to deactivate this option exist, but I expect them > to be very rare, I would prefer to maximize simplicity and maintainability. Agreed with Pablo. Also, once we add a configuration option it becomes delicate to later remove it. Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NUOZW4KRZWPJS4Y77DKD3ZJN24YBHAJ4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks
On Sun, 09 May 2021 02:16:02 - "Jim J. Jewett" wrote: > Antoine Pitrou wrote: > > On Sat, 8 May 2021 02:58:40 + > > Neil Schemenauer nas-pyt...@arctrix.com wrote: > > > > It would be cool if we could mmap the pyc files and have the VM run > > > code without an unmarshal step. > > > What happens if another process mutates or truncates the file while the > > CPython VM is executing code from the mapped file? Crash? > > Why would this be any different than whatever happens now? What happens now is that the pyc file is transferred at once to memory using regular IO. So the chance is really slim that you read invalid data due to concurrent mutation. Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JRK7RPRG2ENUJRTNJU3PD47QSLVOSVXN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks
On 5/9/21 3:00 AM, M.-A. Lemburg wrote: > BTW: For better readability, I'd also not output the lines > for every stack level in the traceback, but just the last one, > since it's usually clear where the call to the next stack > level happens in the upper ones. Usually, sure -- but in the unusual case those carets at every level can save a lot of time and frustration, and that is the goal of this enhancement, yes? I know I have experienced that ambiguity more than once. -- ~Ethan~ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NQPGKRKGOECYWHB2FOOHXEMPV3XTO6CU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 657 – Include Fine Grained Error Locations in Tracebacks
Hi, We have prepared a PEP with our proposal for fine-grained error locations in tracebacks. You can find the PEP here: https://www.python.org/dev/peps/pep-0657/ The discussion is happening in the discourse server: https://discuss.python.org/t/pep-657-include-fine-grained-error-locations-in-tracebacks/8629 To avoid splitting the discussion, *please redirect your comments there* instead of replying to this thread. Thanks! Regards from sunny London, Pablo Galindo Salgado ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ALTWCBWXKBSW5TMQ33GYFXO4SB4TB4QY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks
On 5/9/21 3:00 AM, M.-A. Lemburg wrote: BTW: For better readability, I'd also not output the lines for every stack level in the traceback, but just the last one, since it's usually clear where the call to the next stack level happens in the upper ones. Playing devil's advocate: in the un-usual case, where it may be ambiguous where the call came from, outputting the lines could be a real life-saver. I concede this is rare, //arry/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OTTSTWR4I7AG4J7RJX4PLCGCZO6DVOMZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks
On 08.05.2021 23:55, Gregory P. Smith wrote: > Who non-hypothetically cares about a 22% pyc file size increase? I don't > think > we should be concerned. I'm in favor of always writing them and the 20% size > increase that results in. If pyc size is an issue that should be its own > separate enhancement PEP. When it comes to pyc files there is more data we > may > want to store in the future for performance reasons - I don't see them > shrinking > without an independent effort. > > Caring about additional data retained in memory at runtime makes more sense to > me as ram cost is much greater than storage cost and is paid repeatedly per > process. Storing an additional reference to None on code objects where a > column > information table is perfectly fine. That can be a -X style interpreter > startup > option. It isn't something that needs to impacted by the pyc files. Pass > that > option to the interpreter, and it just discards column info tables on code > objects after loading them or compiling them. If people want to optimize for > a > shared pyc situation with memory mapping techniques, that is also something > that > should be a separate enhancement PEP and not involved here. People writing > code > to use the column information should always check it for None first, that'd be > something we document with the new feature. I do care about both the increase in PYC size as well as the increase in memory usage. When using Python in containers both are relevant and so I'd like an option to switch this whole mechanism off that's independent from optimization settings. This idea is more about debugging during development and doesn't really have much to do with optimization used for production use of Python, so a separate flag or perhaps use of -v would the more intuitive approach. Alternative idea: Create a new file format which supports enhanced debugging. This would include the source code in a indexed format, the AST and mappings between byte code, AST node, lines and columns. Python would then only use and load this file when it needs to print a traceback - much like it does today with the source code. The advantage is that you can add even more useful information for debugging while not making the default code distribution format take more memory (both disk and RAM). BTW: For better readability, I'd also not output the lines for every stack level in the traceback, but just the last one, since it's usually clear where the call to the next stack level happens in the upper ones. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 09 2021) >>> Python Projects, Coaching and Support ...https://www.egenix.com/ >>> Python Product Development ...https://consulting.egenix.com/ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VF3OFSOJIS2PDCGJM2UCELKDHUWYM2HR/ Code of Conduct: http://python.org/psf/codeofconduct/
