Re: [Python-Dev] cffi in stdlib
On Wed, Feb 27, 2013 at 7:57 AM, Eli Bendersky wrote: > > > I read the cffi docs once again and went through some of the examples. I >> > want to divide this to two topics. >> > >> > One is what you call the "ABI" level. IMHO, it's hands down superior to >> > ctypes. Your readdir demo demonstrates this very nicely. I would >> definitely >> > want to see this in the stdlib as an alternative way to interface to C >> > shared objects & DLLs. >> > >> > Two is what you call the "API" level, which is where my opinion becomes >> > mixed. Some things just don't feel right to me: >> > >> > 1. Tying in a C compiler into the flow of a program. I'm not sure >> whether we >> > have precedents for it in the stdlib. Does this work on Windows where >> > libraries and DLLs are usually built with MSVC? >> > >> >> Yes. Precedent in the stdlib is really the C API. All the same rules >> apply (including build and ship a dll). >> > > So would you say that the main use of the API level is provide an > alternative for writing C API code to interface to C libraries. IOW, it's > in competition with Swig? > I'd hardly call it competition. The primary language I interface with is C++ and cffi appears not see that giant elephant in the room (it'd need to use clang for parsing if it were going to do that)... The goal is good, but being C only isn't super exciting to me. Would there be a goal of using cffi to replace C extension module code in the standard library with cffi based versions instead of hand written CPython C API code? If not, why not? (and what does that say about its limitations or practical it is?) -0.5 from me. My inclination is not to add this to the standard library. But even if it were to be added, it sounds like others are coming up with questions and reasons why it isn't yet ready (always the first important step to seriously considering inclusion). -gps ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Planning on removing cache invalidation for file finders
As of right now, importlib keeps a cache of what is in a directory for its file finder instances. It uses mtime on the directory to try and detect when it has changed to know when to refresh the cache. But thanks to mtime granularities of up to a second, it is only a heuristic that isn't totally reliable, especially across filesystems on different OSs. This is why importlib.invalidate_caches() came into being. If you look in our test suite you will see it peppered around where a module is created on the fly to make sure that mtime granularity isn't a problem. But it somewhat negates the point of the mtime heuristic when you have to make this function call regardless to avoid potential race conditions. http://bugs.python.org/issue17330 originally suggested trying to add another heuristic to determine when to invalidate the cache. But even with the suggestion it's still iffy and in no way foolproof. So the current idea is to just drop the invalidation heuristic and go full-blown reliance on calls to importlib.invalidate_caches() as necessary. This makes code more filesystem-agnostic and protects people from hard-to-detect errors when importlib only occasionally doesn't detect new modules (I know it drove me nuts for a while when the buildbots kept failing sporadically and only on certain OSs). I would have just made the change but Antoine wanted it brought up here first to make sure that no one was heavily relying on the current setup. So if you have a good, legitimate reason to keep the reliance on mtime for cache invalidation please speak up. But since the common case will never care about any of this (how many people generate modules on the fly to being with?) and to be totally portable you need to call importlib.invalidate_caches() anyway, it's going to take a lot to convince me to keep it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Fri, 1 Mar 2013 09:32:23 -0500 Barry Warsaw wrote: > > >On the other hand in some ways Jython is sort of like Python on a > >weird virtual OS that lets the real OS bleed through some. This may > >still need to be checked in that way (there's are still checks of >os.name == 'nt'> right?) > > Yeah, but that all old code ;) Hmm, what do you mean? `os.name == 'nt'` is still the proper way to test that we're running on a Windows system (more accurately, over the Windows API). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Merging Jython code into standard Lib [was Re: Python Language Summit at PyCon: Agenda]
On Thu, Feb 28, 2013 at 12:35 PM, Brett Cannon wrote: > > On Thu, Feb 28, 2013 at 3:17 PM, fwierzbi...@gmail.com > wrote: >> >> It would be nice in this particular case if there was a zlib.py that >> imported _zlib -- then it would be easy to shim in Jython's version, >> whether it is written in a .py file or in Java. > > > That should be fine as that is what we already do for accelerator modules > anyway. If you want to work towards having an equivalent of CPython's > Modules/ directory so you can ditch your custom Lib/ modules by treating > your specific code as accelerators I think we can move towards that > solution. Sounds great! I'm betting that implementing PEP 420 on Jython will make mixed Python/Java code easier to deal with, so _zlib.py might just end up living next to our Java code. So deleting Jython's Lib/ may still be an option. -Frank ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2013-02-22 - 2013-03-01) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3877 (+11) closed 25227 (+41) total 29104 (+52) Open issues with patches: 1695 Issues opened (42) == #16121: shlex.shlex.error_leader() reports incorrect line number http://bugs.python.org/issue16121 reopened by r.david.murray #17223: Initializing array.array with unicode type code and buffer seg http://bugs.python.org/issue17223 reopened by sbt #17278: SIGSEGV in _heapqmodule.c http://bugs.python.org/issue17278 opened by maxxedev #17279: Document which named built-in classes can be subclassed http://bugs.python.org/issue17279 opened by terry.reedy #17281: Broken links at pypi http://bugs.python.org/issue17281 opened by lyda #17282: document the defaultTest parameter to unittest.main() http://bugs.python.org/issue17282 opened by chris.jerdonek #17284: create mercurial section in devguide's committing.rst http://bugs.python.org/issue17284 opened by chris.jerdonek #17285: subprocess.check_output incorrectly state that output is alway http://bugs.python.org/issue17285 opened by Baptiste.Lepilleur #17286: Make subprocess handling text output with universal_newlines m http://bugs.python.org/issue17286 opened by Baptiste.Lepilleur #17288: cannot jump from a return after setting f_lineno http://bugs.python.org/issue17288 opened by xdegaye #17289: readline.set_completer_delims() doesn't play well with others http://bugs.python.org/issue17289 opened by bfroehle #17293: uuid.getnode() MAC address on AIX http://bugs.python.org/issue17293 opened by aivarsk #17294: compile-flag for single-execution to return value instead of p http://bugs.python.org/issue17294 opened by Albert.Zeyer #17295: __slots__ on PyVarObject subclass http://bugs.python.org/issue17295 opened by ronaldoussoren #17298: Twisted test failure triggered by change in 2.7 branch http://bugs.python.org/issue17298 opened by glyph #17299: Test cPickle with real files http://bugs.python.org/issue17299 opened by serhiy.storchaka #17301: An in-place version of many bytearray methods is needed http://bugs.python.org/issue17301 opened by gregory.p.smith #17302: HTTP/2.0 - Implementations/Testing efforts http://bugs.python.org/issue17302 opened by karlcow #17305: IDNA2008 encoding missing http://bugs.python.org/issue17305 opened by marten #17306: Improve the way abstract base classes are shown in help() http://bugs.python.org/issue17306 opened by rhettinger #17307: HTTP PUT request Example http://bugs.python.org/issue17307 opened by orsenthil #17308: Dialog.py crashes when putty Window resized http://bugs.python.org/issue17308 opened by harshaap #17309: __bytes__ doesn't work in subclass of int http://bugs.python.org/issue17309 opened by pkoning #17310: Ctypes callbacks shows problem on Windows Python (64bit) http://bugs.python.org/issue17310 opened by Matt.Clarke #17311: use distutils terminology in "PyPI package display" section http://bugs.python.org/issue17311 opened by chris.jerdonek #17312: test_aifc doesn't clean up after itself http://bugs.python.org/issue17312 opened by chris.jerdonek #17314: Stop using imp.find_module() in multiprocessing http://bugs.python.org/issue17314 opened by brett.cannon #17315: test_posixpath doesn't clean up after itself http://bugs.python.org/issue17315 opened by chris.jerdonek #17316: Add Django 1.5 to benchmarks http://bugs.python.org/issue17316 opened by brett.cannon #17317: Benchmark driver should calculate actual benchmark count in -h http://bugs.python.org/issue17317 opened by brett.cannon #17318: xml.sax and xml.dom fetch DTDs by default http://bugs.python.org/issue17318 opened by rsandwick3 #17319: http.server.BaseHTTPRequestHandler send_response_only doesn't http://bugs.python.org/issue17319 opened by karlcow #17320: os.path.abspath in window7, return error http://bugs.python.org/issue17320 opened by xiaowei.py #17322: urllib.request add_header() currently allows trailing spaces ( http://bugs.python.org/issue17322 opened by karlcow #17323: Disable [X refs, Y blocks] ouput in debug builds http://bugs.python.org/issue17323 opened by ezio.melotti #17324: SimpleHTTPServer serves files even if the URL has a trailing s http://bugs.python.org/issue17324 opened by larry #17325: improve organization of the PyPI distutils docs http://bugs.python.org/issue17325 opened by chris.jerdonek #17326: Windows build docs still referring to VS 2008 in 3.3 http://bugs.python.org/issue17326 opened by cito #17327: Add PyDict_GetItemSetDefault() as C-API for dict.setdefault() http://bugs.python.org/issue17327 opened by scoder #17328: Fix reference leak in dict_setdefault() in case of resize fail http://bugs.python.org/issue17328 opened by scoder #17329: Document unittest.SkipTest http://bugs.python.org/issue1732
[Python-Dev] FileCookieJars
Cross-posting from python-ideas due to no response there. Perhaps it's due to a general lack of usage/caring for cookiejar, but figured /someone/'s got to have an opinion about my proposal ;) Note that I've moved my discussion from bug 16942 to 16901 (http://bugs.python.org/issue16901) as they're duplicates and 16901 is more succinct. I've also posted a patch in 16901 implementing my proposal. TL;DR: CookieJar > FileCookieJar > *CookieJar are architecturally broken and this is an attempt to rectify that (and fix a couple bugs along the way). --- Context: http://bugs.python.org/issue16942 (my patch, changing FileCookieJar to be an abc, defining the interfaces for *FileCookieJar). This pertains to Terry's question about whether or not it makes sense that an abstract base class extends a concrete class. After putting in a little thought, he's right. It doesn't make sense. After further thought, I'm relatively confident that the hierarchy as it stands should be changed. Currently what's implemented in the stdlib looks like this: CookieJar | FileCookieJar | | | MozillaCookieJar LWPCookieJar What I'm proposing is that the structure is broken to be the following: FileCookieJarProcessor CookieJar | | | MozillaCookieJarProcessor LWPCookieJarProcessor The intention here is to have processors that operate /on/ a cookiejar object via composition rather than inheritance. This aligns with how urllib.request.HTTPCookieProcessor works (which has the added bonus of cross-module consistency). The only attributes that concrete FileCookieJarProcessor classes touch (at least, in the stdlib) are _cookies and _cookies_lock. I have mixed feelings about whether these should continue to be noted as "non-public" with the _ prefix or not as keeping the _ would break convention of operating on non-public fields, but am unsure of the ramifications of changing them to public. Making this change then allows for FileCookieJar(Processor) to be an abstract base class without inheriting from CookieJar which doesn't make a whole lot of sense from an architecture standpoint. I have yet to see what impact these changes have to the cookiejar extensions at http://wwwsearch.sf.net but plan on doing so if this approach seems sound. This will obviously break backwards compatibility, so I'm not entirely sure what best practice is around that: leave well enough alone even though it might not make sense, keep the old implementations around and deprecate them to be eventually replaced by the processors, or other ideas? -- Demian Brecht http://demianbrecht.github.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Disabling string interning for null and single-char causes segfaults
Hi, I'm working on an automated bug finding tool that I'm trying to apply on the Python interpreter code (version 2.7.3). Because of early prototype limitations, I needed to disable string interning in stringobject.c. More precisely, I modified the PyString_FromStringAndSize and PyString_FromString to no longer check for the null and single-char cases, and create instead a new string every time (I can send the patch if needed). However, after applying this modification, when running "make test" I get a segfault in the test___all__ test case. Before digging deeper into the issue, I wanted to ask here if there are any implicit assumptions about string identity and interning throughout the interpreter implementation. For instance, are two single-char strings having the same content supposed to be identical objects? I'm assuming that it's either this, or some refcount bug in the interpreter that manifests only when certain strings are no longer interned and thus have a higher chance to get low refcount values. Thanks a lot, Stefan Bucur ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Fri, Mar 1, 2013 at 4:41 AM, Stefan Behnel wrote: > Michael Foord, 27.02.2013 17:51: > > PyCon, and the Python Language Summit, is nearly upon us. We have a good > number of people confirmed to attend. If you are intending to come to the > language summit but haven't let me know please do so. > > > > The agenda of topics for discussion so far includes the following: > > > > * A report on pypy status - Maciej and Armin > > * Jython and IronPython status reports - Dino / Frank > > * Packaging (Doug Hellmann and Monty Taylor at least) > > * Cleaning up interpreter initialisation (both in hopes of finding areas > > to rationalise and hence speed things up, as well as making things > > more embedding friendly). Nick Coghlan > > * Adding new async capabilities to the standard library (Guido) > > * cffi and the standard library - Maciej > > * flufl.enum and the standard library - Barry Warsaw > > * The argument clinic - Larry Hastings > > Hi, > > as in the years before, none of the Cython developers is attending the > PyCon-US, so we won't appear that the language summit either. It's a bit > sad that it always takes place at that venue, but I guess there'll just > always be people that can't come to one meeting or the other, so PyCon-US > would just catch the majority. I think it would still be interesting for > many of the attendants to get a status report about Cython, as there seems > to be some confusion and incomplete knowledge about what Cython actually > is, what we have achieved and where we are heading. But maybe the confusion > is large enough to require more than just a little status report to clear > it up. > There are actually two language summits each year: PyCon US and EuroPython. But you are right that the US one is the biggest as it's the easiest to get the most core devs in a single room. Hopefully you can make PyCon US (it's a great conference). And if it's a US issue, it will be in Canada in 2014 and 2015. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Feb 28, 2013, at 08:44 AM, fwierzbi...@gmail.com wrote: >Sorry I meant "is_jython" as a sort of shorthand for a case by case >check. It would be cool if we had a nice set of checks somewhere like >"is_refcounted", etc. Would the sys.implementation area be a good >place for such things? Yep. Unless it proves too unwieldy I suppose. >On the other hand in some ways Jython is sort of like Python on a >weird virtual OS that lets the real OS bleed through some. This may >still need to be checked in that way (there's are still checks of os.name == 'nt'> right?) Yeah, but that all old code ;) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.3): Don't deadlock on a reentrant call.
Hi, On Fri, Mar 1, 2013 at 2:02 PM, raymond.hettinger wrote: > http://hg.python.org/cpython/rev/1920422626a5 > changeset: 82437:1920422626a5 > branch: 3.3 > parent: 82435:43ac02b7e322 > user:Raymond Hettinger > date:Fri Mar 01 03:47:57 2013 -0800 > summary: > Don't deadlock on a reentrant call. this seems to have broken builds without threads. After this commit I get a compile error: $ ./configure --without-threads --with-pydebug && make -j2 [...] ./python -E -S -m sysconfig --generate-posix-vars Could not find platform dependent libraries Consider setting $PYTHONHOME to [:] Could not import runpy module Exception ignored in: 'garbage collection' Traceback (most recent call last): File "/home/wolf/dev/py/py3k/Lib/runpy.py", line 16, in import imp File "/home/wolf/dev/py/py3k/Lib/imp.py", line 23, in import tokenize File "/home/wolf/dev/py/py3k/Lib/tokenize.py", line 28, in import re File "/home/wolf/dev/py/py3k/Lib/re.py", line 124, in import functools File "/home/wolf/dev/py/py3k/Lib/functools.py", line 22, in from dummy_threading import RLock File "/home/wolf/dev/py/py3k/Lib/dummy_threading.py", line 45, in import threading File "/home/wolf/dev/py/py3k/Lib/threading.py", line 6, in from time import sleep as _sleep ImportError: No module named 'time' Fatal Python error: unexpected exception during garbage collection Current thread 0x: make: *** [pybuilddir.txt] Aborted (core dumped) See also: http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/4006 http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%203.3/builds/516 (Also having tests for this change would be nice.) Best Regards, Ezio Melotti ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] High volumes and off topic discussions
Le Thu, 28 Feb 2013 19:17:39 +0200, Maciej Fijalkowski a écrit : > On Thu, Feb 28, 2013 at 4:03 PM, Antoine Pitrou > wrote: > > Le Thu, 28 Feb 2013 13:36:10 +0200, > > Maciej Fijalkowski a écrit : > >> Hi > >> > >> I know this is a hard topic, but python-dev is already incredibly > >> high-volume and dragging discussion off-topic is making following > >> important stuff (while ignoring unimportant stuff) very hard. > >> > >> For example in a recent topic "cffi in stdlib" I find a mail that > >> says "we have to find a sufficiently silly species of snake". It's > >> even funny, but it definitely makes it very hard to follow for > >> those of us who don't read python-dev 24/7. Would it be reasonable > >> for python-dev to generally try to stay on topic (for example if > >> the thread is called "silly species of snakes", I absolutely don't > >> mind people posting there whatever they feel like as long as I'm > >> not expected to read every single message). > > > > I'm afraid you're trying to devise derogatory distinctions regarding > > drifting discussions. > > > > Seriously, yes, I approve of changing the subject line, although I > > forgot to do it this time. > > For the record, you can also read the list through a NNTP gateway > > using Gmane, it can make things easier. > > How does that help with knowing what mails to read what mails not to > read? It doesn't, but at least it won't flood your personal inbox ;-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
Michael Foord, 27.02.2013 17:51: > PyCon, and the Python Language Summit, is nearly upon us. We have a good > number of people confirmed to attend. If you are intending to come to the > language summit but haven't let me know please do so. > > The agenda of topics for discussion so far includes the following: > > * A report on pypy status - Maciej and Armin > * Jython and IronPython status reports - Dino / Frank > * Packaging (Doug Hellmann and Monty Taylor at least) > * Cleaning up interpreter initialisation (both in hopes of finding areas > to rationalise and hence speed things up, as well as making things > more embedding friendly). Nick Coghlan > * Adding new async capabilities to the standard library (Guido) > * cffi and the standard library - Maciej > * flufl.enum and the standard library - Barry Warsaw > * The argument clinic - Larry Hastings Hi, as in the years before, none of the Cython developers is attending the PyCon-US, so we won't appear that the language summit either. It's a bit sad that it always takes place at that venue, but I guess there'll just always be people that can't come to one meeting or the other, so PyCon-US would just catch the majority. I think it would still be interesting for many of the attendants to get a status report about Cython, as there seems to be some confusion and incomplete knowledge about what Cython actually is, what we have achieved and where we are heading. But maybe the confusion is large enough to require more than just a little status report to clear it up. It's also true that many of the topics above aren't really interesting for us, because we just inherit them with CPython, e.g. stdlib changes. Packaging is only relevant as far as it impacts the distribution of binary extensions, and the main changes appear to be outside of that area (which doesn't mean it's not truly wonderful that they are happening, Python packaging has seen a lot of great improvements during the last years and I'm very happy to see it getting even better). Interpreter initialisation would be interesting and Cython could potentially help in some spots here by making code easier to maintain and optimise, for example. We've had this discussion for the importlib bootstrapping and I'm sure there's more that could be done. It's sad to see so much C-level work go into areas that really don't need to be that low-level. I'm not so happy with the argument clinic, but that's certainly also because I'm biased. I've written the argument unpacking code for Cython some years ago, so it's not surprising that I'm quite happy with that and fail to see the need for a totally new DSL *and* a totally new implementation, especially with its mapping to the slowish ParseTuple*() C-API functions. I've also not seen a good argument why the existing Py3 function signatures can't do what the proposed DSL tries to achieve. They'd at least make it clear that the intention is to make things more Python-like, and would at the same time provide the documentation. The topics that would be interesting for us sound more like they'd benefit from a "CPython runtime summit". I really think that it would be beneficial for the CPython developers to learn how we solved problems that they have on their lists or might at least run into in a couple of years, and for us to see if we can't come up with cleaner solutions for problems that CPython currently makes hard to do outside of the core. For example, making C-implemented code Python compatible is actually not trivial and has cost us a lot of investment. Nowadays, CPython is actually further away from that in some areas than Cython, and I don't think it needs to stay that way. It could certainly help both Cython and CPython if CPython gained some of the capabilities for itself that we had to implement ourselves in clean or hacky ways, but always outside of the core. There isn't really a reason why C-implemented parts of CPython must behave all that different from Python implemented parts, why modules must have a different API than other objects, why builtins can't accept keyword arguments, ... These things just get in the way once their camouflage as features is blown up. Another topic is C-level calling between extensions that only see each other through the CPython core. Python call semantics are nice, but also extremely slow compared to C calls. Capsules are simple but slow and static. Implementing a dynamic C calling interface between extensions with a safe way to pass C signatures around and validate them (or adapt to them) on the other side would be much easier with CPython support built into its C function type than trying to do this outside of the core. And there's a huge area of applications for such a feature, especially with the increasing number of tools that do dynamic code generation on the CPython platform. Eventually, this might also become interesting for non-CPythons, as it might provide a way to interface efficiently with CPython extension
