Re: [Python-Dev] head crashing (was: Fwd: [Python-checkins] buildbot warnings in x86 mvlgcc trunk)
In rev 54982 (the first time this crash was seen), I see something which might create a problem. In python/trunk/Modules/posixmodule.c (near line 6300): + PyMem_FREE(mode); Py_END_ALLOW_THREADS Can you call PyMem_FREE() without the GIL held? I couldn't find it documented either way. Of the 3 failures I know of, below is the intersection of the tests that were run prior to crashing: set(['test_threadedtempfile', 'test_cgi', 'test_dircache', 'test_set', 'test_binascii', 'test_imp', 'test_multibytecodec', 'test_weakref', 'test_ftplib', 'test_posixpath', 'test_xmlrpc', 'test_urllibnet', 'test_old_mailbox', 'test_distutils', 'test_site', 'test_runpy', 'test_fork1', 'test_traceback']) n -- On 4/30/07, Neal Norwitz <[EMAIL PROTECTED]> wrote: > This is the third time I've seen a crash on 2 different machines. > This is the first time I noticed this unexplained crash: > > http://python.org/dev/buildbot/all/amd64%20gentoo%20trunk/builds/1983/step-test/0 > > That was at r54982. > > I tried to reproduce this: with a non-debug build, with a debug build, > with valgrind with both types of build. I could never reproduce it. > Valgrind did not report any errors either. > > Here is the third failure: > > http://python.org/dev/buildbot/all/amd64%20gentoo%20trunk/builds/1986/step-test/0 > > The failure below prints: > python: Objects/obmalloc.c:746: PyObject_Malloc: Assertion `bp != > ((void *)0)' failed. > > which probably doesn't really help since the corruption has already > occurred. See > http://python.org/dev/buildbot/all/x86%20mvlgcc%20trunk/builds/497/step-test/0 > > Anyone have ideas what might have caused this? > > n > -- > > -- Forwarded message -- > From: [EMAIL PROTECTED] <[EMAIL PROTECTED]> > Date: Apr 30, 2007 11:17 PM > Subject: [Python-checkins] buildbot warnings in x86 mvlgcc trunk > To: [EMAIL PROTECTED] > > > The Buildbot has detected a new failure of x86 mvlgcc trunk. > Full details are available at: > http://www.python.org/dev/buildbot/all/x86%2520mvlgcc%2520trunk/builds/497 > > Buildbot URL: http://www.python.org/dev/buildbot/all/ > > Build Reason: > Build Source Stamp: [branch trunk] HEAD > Blamelist: georg.brandl > > Build had warnings: warnings test > > Excerpt from the test logfile: > make: *** [buildbottest] Aborted (core dumped) > > sincerely, > -The Buildbot > > ___ > Python-checkins mailing list > [EMAIL PROTECTED] > http://mail.python.org/mailman/listinfo/python-checkins > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.5.1
On 5/1/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> > After doing some research I found that it seems to be impossible to
> > use CreateFile for a file that doesn't have SHARE_READ. I played with
> > different combinations and with FLAG_BACKUP_SEMANTICS and nothing
> > helped. However on Windows there's still a possibility to read
> > attributes: use FindFirstFile. _WIN32_FIND_DATA structure seems to
> > have all the necessary fields (attributes, file times, size and
> > full/short filename), and FindFirstFile doesn't care about sharing at
> > all...
> So what about GetFileAttributesEx? What are the conditions under which
> I can successfully invoke it?
Seems to have the same problems as with CreateFile(...):
// 1.cc
#include
#include
int main(int argc, char** argv)
{
WIN32_FILE_ATTRIBUTE_DATA data;
if(!GetFileAttributesEx("pagefile.sys", GetFileExInfoStandard,
(LPVOID)&data))
{
char buffer[1024];
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
0,
buffer, 1024, NULL);
printf("Error %d: %s\n", GetLastError(), buffer);
return 1;
}
printf("Success\n");
return 0;
}
// 2.cc
#include
#include
int main(int argc, char** argv)
{
WIN32_FIND_DATA data;
if(INVALID_HANDLE_VALUE == FindFirstFile("pagefile.sys", &data))
{
char buffer[1024];
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
0,
buffer, 1024, NULL);
printf("Error %d: %s\n", GetLastError(), buffer);
return 1;
}
printf("Success\n");
return 0;
}
C:\>C:\3\1.exe
Error 32: The process cannot access the file because it is being used
by another process.
C:\>C:\3\2.exe
Success
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.rename on windows
Raghuram Devarakonda wrote: > As a last resort, I > checked cygwin documentation which claims that it's rename() is > POSIX.1 compliant. If I am not mistaken, POSIX.1 does require > atomicity so I am curious how rename() is implemented there. The cygwin implementation of rename goes like this: 1) Try to use MoveFile 2) Try to use MoveFileEx(..., MOVEFILE_REPLACE_EXISTING) 3) Try to unlink destination, then try to use MoveFile And as you say, Cygwin claims it meets POSIX.1. And, POSIX.1 says, "If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing." Clearly, unliking and then calling MoveFile is not atomic. So, cygwin is not being honest here because in these less frequent cases, the rename will not be atomic. Also note, MVCRT only tries step 1 of cygwin's version. Which I believe also suggests that it's the only version that is atomic. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.5.1
Alexey Borzenkov schrieb: > On 5/1/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: >> > After doing some research I found that it seems to be impossible to >> > use CreateFile for a file that doesn't have SHARE_READ. >> So what about GetFileAttributesEx? What are the conditions under which >> I can successfully invoke it? > > Seems to have the same problems as with CreateFile(...): That code only tests it for pagefile.sys. My question was about open handles in general. Both Calvin Spealman and I found that you cannot reproduce the problem when you, in Python 2.5.0, open a file, and then try to os.stat() it - even though, in Python 2.5.0, os.stat() will perform GetFileAttributesEx. So even though we opened the file with not passing any sharing flags, we could still do GetFileAttributesEx on it. I now studied the CRT sources, and it seems that if you use a regular open() call, the CRT will pass FILE_SHARE_READ | FILE_SHARE_WRITE to CreateFile. You would have to use _sopen in the CRT to create any kind of sharing conflict, and that isn't exposed in Python. So I guess we need continue using pagefile.sys as a test case. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 366: Main module explicit relative imports
Brett's PEP 3122 prompted me to finally PEP'ify my proposed solution for the current incompatibility between PEP 328 (absolute imports) and PEP 338 (executing modules as scripts). The only user visible change (other than bug 1510172 going away) would be the presence of a new module level attribute in the main module. Regards, Nick. PEP: 366 Title: Main module explicit relative imports Version: $Revision: 55046 $ Last-Modified: $Date: 2007-05-01 21:13:47 +1000 (Tue, 01 May 2007) $ Author: Nick Coghlan <[EMAIL PROTECTED]> Status: Final Type: Standards Track Content-Type: text/x-rst Created: 1-May-2007 Python-Version: 2.6 Post-History: 1-May-2007 Abstract This PEP proposes a backwards compatible mechanism that permits the use of explicit relative imports from executable modules within packages. Such imports currently fail due to an awkward interaction between PEP 328 and PEP 338 - this behaviour is the subject of at least one open SF bug report (#1510172)[1]. With the proposed mechanism, relative imports will work automatically if the module is executed using the ``-m`` switch. A small amount of boilerplate will be needed in the module itself to allow the relative imports to work when the file is executed by name. Import Statements and the Main Module = (This section is taken from the final revision of PEP 338) The release of 2.5b1 showed a surprising (although obvious in retrospect) interaction between PEP 338 and PEP 328 - explicit relative imports don't work from a main module. This is due to the fact that relative imports rely on ``__name__`` to determine the current module's position in the package hierarchy. In a main module, the value of ``__name__`` is always ``'__main__'``, so explicit relative imports will always fail (as they only work for a module inside a package). Investigation into why implicit relative imports *appear* to work when a main module is executed directly but fail when executed using ``-m`` showed that such imports are actually always treated as absolute imports. Because of the way direct execution works, the package containing the executed module is added to sys.path, so its sibling modules are actually imported as top level modules. This can easily lead to multiple copies of the sibling modules in the application if implicit relative imports are used in modules that may be directly executed (e.g. test modules or utility scripts). For the 2.5 release, the recommendation is to always use absolute imports in any module that is intended to be used as a main module. The ``-m`` switch already provides a benefit here, as it inserts the current directory into ``sys.path``, instead of the directory contain the main module. This means that it is possible to run a module from inside a package using ``-m`` so long as the current directory contains the top level directory for the package. Absolute imports will work correctly even if the package isn't installed anywhere else on sys.path. If the module is executed directly and uses absolute imports to retrieve its sibling modules, then the top level package directory needs to be installed somewhere on sys.path (since the current directory won't be added automatically). Here's an example file layout:: devel/ pkg/ __init__.py moduleA.py moduleB.py test/ __init__.py test_A.py test_B.py So long as the current directory is ``devel``, or ``devel`` is already on ``sys.path`` and the test modules use absolute imports (such as ``import pkg.moduleA`` to retrieve the module under test, PEP 338 allows the tests to be run as:: python -m pkg.test.test_A python -m pkg.test.test_B Rationale for Change In rejecting PEP 3122 (which proposed a higher impact solution to this problem), Guido has indicated that he still isn't particularly keen on the idea of executing modules inside packages as scripts [2]. Despite these misgivings he has previously approved the addition of the ``-m`` switch in Python 2.4, and the ``runpy`` module based enhancements described in PEP 338 for Python 2.5. The philosophy that motivated those previous additions (i.e. access to utility or testing scripts without needing to worry about name clashes in either the OS executable namespace or the top level Python namespace) is also the motivation behind fixing what I see as a bug in the current implementation. This PEP is intended to provide a solution which permits explicit relative imports from main modules, without incurring any significant costs during interpreter startup or normal module import. Proposed Solution = The heart of the proposed solution is a new module attribute ``__package_name__``. This attribute will be defined only in the main module (i.e. modules where ``__name__ == "__main__"``). For a directly executed main module, this attribute will b
Re: [Python-Dev] Python 2.5.1
On 5/1/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > That code only tests it for pagefile.sys. My question was about open > handles in general. Both Calvin Spealman and I found that you cannot > reproduce the problem when you, in Python 2.5.0, open a file, and then > try to os.stat() it - even though, in Python 2.5.0, os.stat() will > perform GetFileAttributesEx. So even though we opened the file with > not passing any sharing flags, we could still do GetFileAttributesEx > on it. > > I now studied the CRT sources, and it seems that if you use a regular > open() call, the CRT will pass FILE_SHARE_READ | FILE_SHARE_WRITE to > CreateFile. You would have to use _sopen in the CRT to create any > kind of sharing conflict, and that isn't exposed in Python. Wow, I'm very sorry, I didn't realize how much special pagefile.sys and hiberfil.sys are. As it turns out, even if you create a file with no sharing allowed, you can still open it with backup semantics in other processes, and thus can use GetFileAttributesEx, GetFileTime, etc. The file pagefile.sys seems almost magical then, I don't understand how it's opened to behave like that. The difference is also immediately visible if you try to open Properties of pagefile.sys, you won't even see Security tab there (even when I create file something.txt and then remove all ACLs, including SYSTEM, I can't access the file, but I can see Security tab and can grant myself permissions back), it looks like all kinds of opening that file are denied. Maybe this is a special security feature, so that no process could access swapped pages (otherwise it could be possible with backup semantics). Thus you can't access the file itself, you can only access containing directory. > So I guess we need continue using pagefile.sys as a test case. Seems to be true, it's just maybe it shouldn't be hardcoded to C:\ There's REG_MULTI_SZ PagingFiles in "HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management", btw. The format seems to be "filename minmbsize maxmbsize" for every line. Best regards, Alexey. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.5.1
Hm, I fail to see the importance of a special regression test for that peculiar file then with its special magical OS properties. Why not focus our attention on real, user generated files?. -Original Message- Wow, I'm very sorry, I didn't realize how much special pagefile.sys and hiberfil.sys are. As it turns out, even if you create a file with no sharing allowed, you can still open it with backup semantics in other processes, and thus can use GetFileAttributesEx, GetFileTime, etc. The file pagefile.sys seems almost magical then, I don't understand how it's opened to behave like that. The difference is also immediately visible if you try to open Properties of pagefile.sys, you won't even see Security tab there (even when I create file something.txt and then remove all ACLs, including SYSTEM, I can't access the file, but I can see Security tab and can grant myself permissions back), it looks like all kinds of opening that file are denied. Maybe this is a special security feature, so that no process could access swapped pages (otherwise it could be possible with backup semantics). Thus you can't access the file itself, you can only access containing directory. > So I guess we need continue using pagefile.sys as a test case. Seems to be true, it's just maybe it shouldn't be hardcoded to C:\ There's REG_MULTI_SZ PagingFiles in "HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management", btw. The format seems to be "filename minmbsize maxmbsize" for every line. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.5.1
Kristján Valur Jónsson schrieb: > Hm, I fail to see the importance of a special regression test for that > peculiar file then with its special magical OS properties. Why not focus > our attention on real, user generated files?. Because real users really had real problems with this very real file, or else they had not reported that as a real bug, really. Are you proposing to unfix the bug? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.rename on windows
On 5/1/07, Scott Dial <[EMAIL PROTECTED]> wrote: > The cygwin implementation of rename goes like this: > > 1) Try to use MoveFile > 2) Try to use MoveFileEx(..., MOVEFILE_REPLACE_EXISTING) > 3) Try to unlink destination, then try to use MoveFile > > And as you say, Cygwin claims it meets POSIX.1. And, POSIX.1 says, "If > newpath already exists it will be atomically replaced (subject to > a few conditions; see ERRORS below), so that there is no point at which > another process attempting to access newpath will find it missing." > Clearly, unliking and then calling MoveFile is not atomic. So, cygwin is > not being honest here because in these less frequent cases, the rename > will not be atomic. You are right. I just checked cygwin's rename() code and it is convincing enough for me to withdraw the patch. Thanks for all the comments. Raghu ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.5.1
On 5/1/07, Kristján Valur Jónsson <[EMAIL PROTECTED]> wrote: > Hm, I fail to see the importance of a special regression test for that > peculiar file then with its special magical OS properties. Why not focus > our attention on real, user generated files?. (Try to stick to the posting conventions and reply under the actual segments you respond to.) Not all the user generated files are directly from python. Consider all the extension libraries that can do anything they want opening files on lower levels. For example, database files are likely to have different sharing flags than the default. I'm not sure if sqlite, for example, may or may not have such problems. Martin: Would tests that use ctypes do do the open directly be acceptable ways of solving this? > -Original Message- > > Wow, I'm very sorry, I didn't realize how much special pagefile.sys > and hiberfil.sys are. As it turns out, even if you create a file with > no sharing allowed, you can still open it with backup semantics in > other processes, and thus can use GetFileAttributesEx, GetFileTime, > etc. The file pagefile.sys seems almost magical then, I don't > understand how it's opened to behave like that. The difference is also > immediately visible if you try to open Properties of pagefile.sys, you > won't even see Security tab there (even when I create file > something.txt and then remove all ACLs, including SYSTEM, I can't > access the file, but I can see Security tab and can grant myself > permissions back), it looks like all kinds of opening that file are > denied. Maybe this is a special security feature, so that no process > could access swapped pages (otherwise it could be possible with backup > semantics). Thus you can't access the file itself, you can only access > containing directory. > > > So I guess we need continue using pagefile.sys as a test case. > > Seems to be true, it's just maybe it shouldn't be hardcoded to C:\ > There's REG_MULTI_SZ PagingFiles in > "HKLM\System\CurrentControlSet\Control\Session Manager\Memory > Management", btw. The format seems to be "filename minmbsize > maxmbsize" for every line. > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.5.1
> Would tests that use ctypes do do the open directly be acceptable ways > of solving this? If it solves it - sure. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New operations in Decimal
On 27 Apr, 2007, at 20:39, Facundo Batista wrote: - and (and), or (or), xor (xor) [CD]: Takes two logical operands, the result is the logical operation applied between each digit. "and" and "or" are keywords, you can't have methods with those names: >>> def and(l, r): pass File "", line 1 def and(l, r): pass ^ SyntaxError: invalid syntax >>> Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP index out of date, and work-around
There seems to be an issue with the PEP index: http://python.org/dev/peps/ lists PEP 3122 as the last PEP (not counting PEP 3141 which is deliberately out of sequence). As a work-around, an up to date index is here: http://python.org/dev/peps/pep-/ PEPs 3123-3128 are alive and well and reachable via this index. One of the webmasters will look into this tonight. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New operations in Decimal
Ronald Oussoren <[EMAIL PROTECTED]> wrote: > On 27 Apr, 2007, at 20:39, Facundo Batista wrote: > > > - and (and), or (or), xor (xor) [CD]: Takes two logical operands, the > > result is the logical operation applied between each digit. > > "and" and "or" are keywords, you can't have methods with those names: Am I losing my marbles, or is this a proposal to add the logical operations to FLOATING-POINT? I may have missed a previous posting, in which case I apologise, but this is truly mind-boggling. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] head crashing (was: Fwd: [Python-checkins] buildbot warnings in x86 mvlgcc trunk)
On 5/1/07, Neal Norwitz <[EMAIL PROTECTED]> wrote: In rev 54982 (the first time this crash was seen), I see something which might create a problem. In python/trunk/Modules/posixmodule.c (near line 6300): + PyMem_FREE(mode); Py_END_ALLOW_THREADS The PyMem_MALLOC call that creates 'mode' is also called without explicitly holding the GIL. Can you call PyMem_FREE() without the GIL held? I couldn't find it documented either way. I believe the GIL does not need to be held, but obviously Tim or someone with more memory experience should step in to say definitively. If you look at Include/pymem.h, PyMem_FREE gets defined as PyObject_FREE in a debug build. PyObject_Free is defined at _PyObject_DebugFree. That function checks that the memory has not been written with the debug bit pattern and then calls PyObject_Free. That call just sticks the memory back into pymalloc's memory pool which is implemented without using any Python objects. In other words no Python objects are used in pymalloc (to my knowledge) and thus is safe to use without the GIL. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] head crashing
> I believe the GIL does not need to be held, but obviously Tim or someone > with more memory experience should step in to say definitively. > > If you look at Include/pymem.h, PyMem_FREE gets defined as PyObject_FREE > in a debug build. PyObject_Free is defined at _PyObject_DebugFree. > That function checks that the memory has not been written with the debug > bit pattern and then calls PyObject_Free. That call just sticks the > memory back into pymalloc's memory pool which is implemented without > using any Python objects. > > In other words no Python objects are used in pymalloc (to my knowledge) This is also what I found. > and thus is safe to use without the GIL. but I got to a different conclusion. If it really goes through the pymalloc pool (obmalloc), then it must hold the GIL while doing so. obmalloc itself is not thread-safe, and relies on the GIL for thread-safety. In release mode, PyMEM_FREE goes directly to free, which is thread-safe. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] head crashing (was: Fwd: [Python-c heckins] buildbot warnings in x86 mvlgcc trunk)
Neal Norwitz gmail.com> writes: > > Can you call PyMem_FREE() without the GIL held? I couldn't find it > documented either way. Nope. See comments at the top of Python/pystate.c. Cheers, mwh ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] head crashing
> but I got to a different conclusion. If it really goes through the > pymalloc pool (obmalloc), then it must hold the GIL while doing so. > obmalloc itself is not thread-safe, and relies on the GIL for > thread-safety. > > In release mode, PyMEM_FREE goes directly to free, which is thread- > safe. Yes. It is quite unfortunate how PyMem_* gets redirected to the PyObject_* functions in debug builds. Even worse is how PyObject_Malloc gets #defined to PyObject_DebugMalloc for debug builds, changing linkage of modules. But that is a different matter. One thing I'd like to point out however, is that it is quite unnecessary for the PyObject_DebugMalloc() functions to lie on top of PyObject_Malloc() They can just call malloc() etc. directly, since in debug builds the performance benefit of the block allocator is moot. I'd suggest to keep the debug functions as a thin layer on top of malloc to do basic testing. I'd even suggest that we reverse things, and move the debug library to pymem.c. This would keep the debug functionalty threadsafe on top of regular malloc, rather than wrapping it in there with the non-threadsafe object allocator. We would then have void *PyMem_DebugMalloc() /* layers malloc /* void *PyMem_Malloc() /* calls PyMem_MALLOC */ #ifndef _DEBUG #define PyMem_MALLOC malloc #else #define PyMem_MALLOC PyMem_DebugMalloc #endif PyObject_Malloc() would then just call PyMem_DebugMalloc in DEBUG builds. The reason I have opinions on this is that at CCP we have spent considerable effort on squeezing our own veneer functions into the memory allocators, both for the PyMem ones and PyObject. And the structure of the macros and their interconnectivity really doesn't make it easy. We ended up creating a set of macros like PyMem_MALLOC_INNER() and ease our functions between the MALLOC and the INNER. I'll try to show you the patch one day which is a reasonable attempt at a slight reform in the structure of these memory APIs. Perhaps something for Py3K. Kristjan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New operations in Decimal
Ronald Oussoren wrote: >> - and (and), or (or), xor (xor) [CD]: Takes two logical operands, the >> result is the logical operation applied between each digit. > > "and" and "or" are keywords, you can't have methods with those names: You're right. I'll name them logical_and, logical_or, and logical_xor. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New operations in Decimal
Nick Maclaren wrote: > Am I losing my marbles, or is this a proposal to add the logical > operations to FLOATING-POINT? Sort of. This is a proposal to keep compliant with the General Decimal Arithmetic Specification, as we promised. http://www2.hursley.ibm.com/decimal/ Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New operations in Decimal
Facundo Batista <[EMAIL PROTECTED]> wrote: > > > Am I losing my marbles, or is this a proposal to add the logical > > operations to FLOATING-POINT? > > Sort of. This is a proposal to keep compliant with the General Decimal > Arithmetic Specification, as we promised. > > http://www2.hursley.ibm.com/decimal/ Or, more precisely: http://www2.hursley.ibm.com/decimal/damisc.html All right. Neither you nor I have lost our marbles, but the authors of that assuredly did. It's totally insane. And implementing it for a software emulation of that specification built on top of a twos complement binary integer model is insanity squared. But promises are promises and mere insanity is not in itself an obstacle to political success I shall attempt to forget that I ever asked the question :-) Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] head crashing
Kristján> I'd suggest to keep the debug functions as a thin layer on top Kristján> of malloc to do basic testing. But then you would substantially change the memory access behavior of the program in a debug build, that is, more than it is already changed by the fact that you have changed the memory layout of Python objects. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] head crashing (was: Fwd: [Python-checkins] buildbot warnings in x86 mvlgcc trunk)
[Neal Norwitz] >> In rev 54982 (the first time this crash was seen), I see something >> which might create a problem. In python/trunk/Modules/posixmodule.c >> (near line 6300): >> >> + PyMem_FREE(mode); >>Py_END_ALLOW_THREADS Shouldn't do that. [Brett Cannon] > The PyMem_MALLOC call that creates 'mode' is also called without explicitly > holding the GIL. Or that ;-) >> Can you call PyMem_FREE() without the GIL held? I couldn't find it >> documented either way. > I believe the GIL does not need to be held, but obviously Tim or someone > with more memory experience should step in to say definitively. The GIL should be held. The relevant docs are in the Python/C API manual, section "8.1 Thread State and the Global Interpreter Lock": Therefore, the rule exists that only the thread that has acquired the global interpreter lock may operate on Python objects or call Python/C API functions. PyMem_XYZ is certainly a "Python/C API function". There are functions you can call without holding the GIL, and section 8.1 intends to give an exhaustive list of those. These are functions that can't rely on the GIL, like PyEval_InitThreads() (which /creates/ the GIL), and various functions that create and destroy thread and interpreter state. > If you look at Include/pymem.h, PyMem_FREE gets defined as PyObject_FREE in > a debug build. PyObject_Free is defined at _PyObject_DebugFree. That > function checks that the memory has not been written with the debug bit > pattern and then calls PyObject_Free. That call just sticks the memory back > into pymalloc's memory pool which is implemented without using any Python > objects. But pymalloc's pools have a complex internal structure of their own, and cannot be mucked with safely by multiple threads simultaneously. > In other words no Python objects are used in pymalloc (to my knowledge) and > thus is safe to use without the GIL. Nope. For example, if two threads simultaneously try to free objects in the same obmalloc size class, there are a number of potential thread-race disasters in linking the objects into the same size-class chain. In a release build this doesn't matter, since PyMem_XYZ map directly to the platform malloc/realloc/free, and so inherit the thread safety (or lack thereof) of the platform C implementations. If it's necessary to do malloc/free kinds of things without holding the GIL, then the platform malloc/free must be called directly. Perhaps that's what posixmodule.c wants to do in this case. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] head crashing (was: Fwd: [Python-checkins] buildbot warnings in x86 mvlgcc trunk)
On 5/1/07, Tim Peters <[EMAIL PROTECTED]> wrote: [Neal Norwitz] >> In rev 54982 (the first time this crash was seen), I see something >> which might create a problem. In python/trunk/Modules/posixmodule.c >> (near line 6300): >> >> + PyMem_FREE(mode); >>Py_END_ALLOW_THREADS Shouldn't do that. [Brett Cannon] > The PyMem_MALLOC call that creates 'mode' is also called without explicitly > holding the GIL. Or that ;-) Luckily I misread the code so it doesn't do that boo-boo. Can you call PyMem_FREE() without the GIL held? I couldn't find it >> documented either way. > I believe the GIL does not need to be held, but obviously Tim or someone > with more memory experience should step in to say definitively. The GIL should be held. The relevant docs are in the Python/C API manual, section "8.1 Thread State and the Global Interpreter Lock": Therefore, the rule exists that only the thread that has acquired the global interpreter lock may operate on Python objects or call Python/C API functions. PyMem_XYZ is certainly a "Python/C API function". There are functions you can call without holding the GIL, and section 8.1 intends to give an exhaustive list of those. These are functions that can't rely on the GIL, like PyEval_InitThreads() (which /creates/ the GIL), and various functions that create and destroy thread and interpreter state. > If you look at Include/pymem.h, PyMem_FREE gets defined as PyObject_FREE in > a debug build. PyObject_Free is defined at _PyObject_DebugFree. That > function checks that the memory has not been written with the debug bit > pattern and then calls PyObject_Free. That call just sticks the memory back > into pymalloc's memory pool which is implemented without using any Python > objects. But pymalloc's pools have a complex internal structure of their own, and cannot be mucked with safely by multiple threads simultaneously. Ah, OK. That makes sense. Glad I pointed out my ignorance then. =) -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Super PEP
On 4/29/07, Calvin Spealman <[EMAIL PROTECTED]> wrote: > Draft Attempt Number Duo: > > PEP: XXX > Title: New Super Checked in as PEP 3133. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Super PEP
Georg Brandl has just checked this PEP in as 367. I had submitted it to the [EMAIL PROTECTED] address, per the policy documentation. Sorry if I subverted some policy order, or was non-vocal about it. I didn't realize anyone else would check it in. On 5/1/07, Guido van Rossum <[EMAIL PROTECTED]> wrote: > On 4/29/07, Calvin Spealman <[EMAIL PROTECTED]> wrote: > > Draft Attempt Number Duo: > > > > PEP: XXX > > Title: New Super > > Checked in as PEP 3133. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Super PEP
Totally my screwup. I'll discard 3133. On 5/1/07, Calvin Spealman <[EMAIL PROTECTED]> wrote: > Georg Brandl has just checked this PEP in as 367. I had submitted it > to the [EMAIL PROTECTED] address, per the policy documentation. Sorry if > I subverted some policy order, or was non-vocal about it. I didn't > realize anyone else would check it in. > > On 5/1/07, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > On 4/29/07, Calvin Spealman <[EMAIL PROTECTED]> wrote: > > > Draft Attempt Number Duo: > > > > > > PEP: XXX > > > Title: New Super > > > > Checked in as PEP 3133. > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > > -- > Read my blog! I depend on your acceptance of my opinion! I am interesting! > http://ironfroggy-code.blogspot.com/ > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
