Re: [Python-Dev] PyFAQ: thread-safe interpreter operations

[Nick Coghlan]

Martin v. Löwis wrote:
> I personally consider it "good style" to rely on implementation details
> of CPython;

Is there a 'do not' missing somewhere in there?

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] DRAFT: python-dev summary for 2006-10-16 to 2006-10-31

[Steven Bethard]

Here's the summary for the second half of October.  Comments and
corrections welcome as always, especially on that extended buffer
protocol / binary format specifier discussion which was a little
overwhelming. ;-)


=
Announcements
=

--
Roundup to replace SourceForge tracker
--

Roundup_ has been named as the official replacement for the
SourceForge_ issue tracker. Thanks go out to the new volunteer admins,
Paul DuBois, Michael Twomey, Stefan Seefeld, and Erik Forsberg, and
also to `Upfront Systems`_ who will be hosting the tracker. If you'd
like to provide input on what the new tracker should do, please join
the `tracker-discuss mailing list`_.

.. _SourceForge: http://www.sourceforge.net/
.. _Roundup: http://roundup.sourceforge.net/
.. _Upfront Systems: http://www.upfrontsystems.co.za/
.. _tracker-discuss mailing list:
http://mail.python.org/mailman/listinfo/tracker-discuss

Contributing threads:

- `PSF Infrastructure has chosen Roundup as the issue tracker for
Python development
`__
- `Status of new issue tracker
`__


=
Summaries
=

---
The buffer protocol and communicating binary format information
---

Travis E. Oliphant presented a pre-PEP for adding a standard way to
describe the shape and intended types of binary-formatted data. It was
accompanied by a pre-PEP for extending the buffer protocol to handle
such shapes and types. Under the proposal, a new ``datatype`` object
would describe binary-formatted data with an API like::

datatype((float, (3,2))
# describes a 3*2*8=48 byte block of memory that should be interpreted
# as 6 doubles laid out as arr[0,0], arr[0,1], ... a[2,0], a[1,2]

datatype([( ([1,2],'coords'), 'f4', (3,6)), ('address', 'S30')])
# describes the structure
# float coords[3*6]   /* Has [1,2] associated with this field */
# char  address[30]

Alexander Belopolsky provided a nice example of why you might want to
extend the buffer protocol along these lines. Currently, there's not
much you can do with a basic buffer object. If you want to pass it to
numpy_, you have to provide the type and shape information yourself::

>>> b = buffer(array('d', [1,2,3]))
>>> numpy.ndarray(shape=(3,), dtype=float, buffer=b)
array([ 1.,  2.,  3.])

By extending the buffer protocol appropriately so that the necessary
information can be provided, you should be able to pass the buffer
directly to numpy_ and have it understand the format itself::

>>> numpy.array(b)

People were uncomfortable with the many ``datatype`` variants -- the
constructor accepted types, strings, lists or dicts, each of which
could specify the structure in a different way. Also, a number of
people questioned why the existing ``ctypes`` mechanisms for
describing binary data couldn't be used instead, particularly since
``ctypes`` could already describe things like function pointers and
recursive types, which the pre-PEP could not. Travis said he was
looking for a way to unify the data formats of all the ``array``,
``struct``, ``numpy`` and ``ctypes`` modules, and felt like using the
``ctypes`` approach was too verbose for use in the other modules. In
particular, he felt like the ``ctypes`` use of type objects as
binary-format specifiers was problematic because type objects were
harder to manipulate at the C level.

The discussion continued on into the next fortnight.

.. _numpy:


Contributing threads:

- `PEP: Adding data-type objects to Python
`__
- `PEP: Extending the buffer protocol to share array information.
`__


The "lazy strings" patch


Discussion continued on Larry Hastings `lazy strings patch`_ that
would have delayed until necessary the evaluation of some string
operations, like concatenation and slicing. With his patch, repeated
string concatenation could be used instead of the standard ``.join()``
idiom, and slices which were never used would never be rendered.
Discussions of the patch showed that people were concerned about
memory increases when a small slice of a very large string kept the
large string around in memory. People also felt like a stronger
motivation was necessary to justify complicating the string
representation so much. Larry was pointed to some `code that his patch
would break`_, which was using ``ob_sval`` directly instead of calling
``PyString_AS_STRING()`` like it was supposed to. He was also referred
to the `Python 3000 list`_ where the recent discussions of `string
views`_ would be relevant, a

Re: [Python-Dev] PyFAQ: thread-safe interpreter operations

[Martin v. Löwis]

Nick Coghlan schrieb:
> Martin v. Löwis wrote:
>> I personally consider it "good style" to rely on implementation details
>> of CPython;
> 
> Is there a 'do not' missing somewhere in there?

No - I really mean it. I can find nothing wrong with people relying on
reference counting to close files, for example. It's a property of
CPython, and not guaranteed in other Python implementations - yet it
works in a well-defined way in CPython. Code that relies on that feature
is not portable, but portability is only one goal in software
development, and may be irrelevant for some projects.

Likewise, I see nothing wrong with people relying on .append on a list
working "correctly" when used from two threads, even though the language
specification does not guarantee that property.

Similarly, it's fine when people rely on the C type "int" to have
32-bits when used with gcc on x86 Linux. The C standard makes that
implementation-defined, but this specific implementation made a
choice that you can rely on.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Weekly Python Patch/Bug Summary

[Kurt B. Kaiser]

Patch / Bug Summary
___

Patches :  406 open (-10) /  3479 closed (+16) /  3885 total ( +6)
Bugs:  931 open ( +1) /  6349 closed (+16) /  7280 total (+17)
RFE :  245 open ( +1) /   244 closed ( +0) /   489 total ( +1)

New / Reopened Patches
__

Logging Module - followfile patch  (2006-11-17)
   http://python.org/sf/1598415  reopened by  cjschr

Logging Module - followfile patch  (2006-11-17)
   http://python.org/sf/1598415  opened by  chads

Logging Module - followfile patch  (2006-11-17)
CLOSED http://python.org/sf/1598426  opened by  chads

mailbox.py: check that os.fsync is available before using it  (2006-11-19)
   http://python.org/sf/1599256  opened by  David Watson

CodeContext - Improved text indentation  (2005-11-21)
CLOSED http://python.org/sf/1362975  reopened by  taleinat

TCPServer option to bind and activate  (2006-11-20)
   http://python.org/sf/1599845  opened by  Peter Parente

__bool__ instead of __nonzero__  (2006-11-21)
   http://python.org/sf/1600346  opened by  ganges master

1572210 doc patch  (2006-11-21)
   http://python.org/sf/1600491  opened by  Jim Jewett

Patches Closed
__

Logging Module - followfile patch  (2006-11-17)
   http://python.org/sf/1598426  closed by  gbrandl

tkSimpleDialog.askstring()  Tcl/Tk-8.4 lockup  (2006-08-11)
   http://python.org/sf/1538878  closed by  loewis

tkSimpleDialog freezes when apply raises exception  (2006-11-11)
   http://python.org/sf/1594554  closed by  loewis

Tix: subwidget names (bug #1472877)  (2006-10-25)
   http://python.org/sf/1584712  closed by  loewis

better error msgs for some TypeErrors  (2006-10-29)
   http://python.org/sf/1586791  closed by  gbrandl

Auto Complete module for IDLE  (2005-11-19)
   http://python.org/sf/1361016  closed by  loewis

Add BLANK_LINE to token.py  (2004-11-20)
   http://python.org/sf/1070218  closed by  loewis

improve embeddability of python  (2003-11-25)
   http://python.org/sf/849278  closed by  loewis

Extend struct.unpack to produce nested tuples  (2003-11-23)
   http://python.org/sf/847857  closed by  loewis

Iterating closed StringIO.StringIO  (2005-11-18)
   http://python.org/sf/1359365  closed by  loewis

urllib reporthook could be more informative  (2003-11-26)
   http://python.org/sf/849407  closed by  loewis

xmlrpclib - marshalling new-style classes.  (2004-11-20)
   http://python.org/sf/1070046  closed by  loewis

CodeContext - Improved text indentation  (2005-11-21)
   http://python.org/sf/1362975  closed by  loewis

Implementation of PEP 3102 Keyword Only Argument  (2006-08-30)
   http://python.org/sf/1549670  closed by  gvanrossum

readline does not need termcap  (2004-12-01)
   http://python.org/sf/1076826  closed by  loewis

Make cgi.py use logging module  (2004-12-06)
   http://python.org/sf/1079729  closed by  loewis

New / Reopened Bugs
___

The empty set is a subset of the empty set  (2006-11-17)
CLOSED http://python.org/sf/1598166  opened by  Andreas Kloeckner

subprocess.py: O(N**2) bottleneck  (2006-11-16)
   http://python.org/sf/1598181  opened by  Ralf W. Grosse-Kunstleve

import curses fails  (2006-11-17)
CLOSED http://python.org/sf/1598357  opened by  thorvinrhuebarb

Misspelled submodule names for email module.  (2006-11-17)
CLOSED http://python.org/sf/1598361  opened by  Dmytro O. Redchuk

ctypes Structure allows recursive definition  (2006-11-17)
   http://python.org/sf/1598620  opened by  Lenard Lindstrom

csv library does not handle '\x00'  (2006-11-18)
CLOSED http://python.org/sf/1599055  opened by  Stephen Day

--disable-sunaudiodev --disable-tk does not work  (2006-10-17)
CLOSED http://python.org/sf/1579029  reopened by  thurnerrupert

mailbox: other programs' messages can vanish without trace  (2006-11-19)
   http://python.org/sf/1599254  opened by  David Watson

htmlentitydefs.entitydefs assumes Latin-1 encoding  (2006-11-19)
CLOSED http://python.org/sf/1599325  opened by  Erik Demaine

SSL-ed sockets don't close correct?  (2004-06-24)
   http://python.org/sf/978833  reopened by  arigo

Segfault on bsddb.db.DB().type()  (2006-11-20)
CLOSED http://python.org/sf/1599782  opened by  Rob Sanderson

problem with socket.gethostname documentation  (2006-11-20)
CLOSED http://python.org/sf/1599879  opened by  Malte Helmert

Immediate Crash on Open  (2006-11-20)
   http://python.org/sf/1599931  opened by  Farhymn

mailbox: Maildir.get_folder does not inherit factory  (2006-11-21)
   http://python.org/sf/1600152  opened by  Tetsuya Takatsuru

[PATCH] Quitting The Interpreter  (2006-11-20)
CLOSED http://python.org/sf/1600157  opened by  Chris Carter

Tix ComboBox entry is blank when not editable  (2006-11-21)
   http://python.org/sf/1600182  opened by  Tim Wegener

--enable-shared links extensions to libpython statically  (2006-11-22)
   http://python.org/sf/1600860  opened by  Marien Zw

[Python-Dev] DRAFT: python-dev summary for 2006-11-01 to 2006-11-15

[Steven Bethard]

Here's the summary for the first half of November. Try not to spend it
all in one place! ;-)

As always, corrections and comments are greatly appreciated.


=
Announcements
=

--
Python 2.5 malloc families
--

Just a reminder that if you find your extension module is crashing
with Python 2.5 in malloc/free, there is a high chance that you have a
mismatch in malloc "families". Unlike previous versions, Python 2.5 no
longer allows sloppiness here -- if you allocate with the ``PyMem_*``
functions, you must free with the ``PyMem_*`` functions, and
similarly, if you allocate with the ``PyObject_*`` functions, you must
free with the ``PyObject_*`` functions.

Contributing thread:

- `2.5 portability problems
`__


=
Summaries
=

--
Path algebra and related functions
--

Mike Orr started work on a replacement for `PEP 355`_ that would
better group the path-related functions currently in ``os``,
``os.path``, ``shutil`` and other modules. He proposed to start with a
`directory-tuple Path class`_ that would have allowed code like::

# equivalent to
# os.path.join(os.path.dirname(os.path.dirname(__FILE__)), "lib")
os.path.Path(__FILE__)[:-2] + "lib"

where a Path object would act like a tuple of directories, and could
be easily sliced and reordered as such.

As an alternative, glyph proposed using `Twisted's filepath module`_
which was already being used in a large body of code. He showed some
common pitfalls, like that the existence on Windows of "CON" and "NUL"
in *every* directory can make paths invalid, and indicated how
FilePath solved these problems.

Fredrik Lundh suggested a reorganization where functions that
manipulate path *names* would reside in ``os.path``, and functions
that manipulate *objects* identified by a path would reside in ``os``.
The ``os.path`` module would gain a path wrapper object, which would
allow "path algebra" manipulations, e.g. ``path1 + path2``. The ``os``
module would gain some of the ``os.path`` and ``shutil`` functions
that were manipulating real filesystem objects and not just the path
names. Most people seemed to like this approach, because it correctly
targeted the "algebraic" features at the areas where chained
operations were most common: path name operations, not filesystem
operations.

Some of the conversation moved on to the `Python 3000 list`_.

.. _PEP 355: http://www.python.org/dev/peps/pep-0355/
.. _directory-tuple Path class: http://wiki.python.org/moin/AlternativePathClass
.. _Twisted's filepath module:
http://twistedmatrix.com/trac/browser/trunk/twisted/python/filepath.py
.. _Python 3000 list: http://mail.python.org/mailman/listinfo/python-3000

Contributing threads:

- `Path object design
`__
- `Mini Path object
`__
- `[Python-3000] Mini Path object
`__

--
Replacing urlparse
--

A few more bugs in ``urlparse`` were turned up, and `earlier
discussions about replacing urlparse`_ were briefly revisited. Paul
Jimenez asked about `uriparse module`_ and was told that due to the
constant problems with ``urlparse``, people were concerned about
including the "incorrect" library again, so requirements were a little
stringent. Martin v. Löwis gave him some guidance on a few specific
points, and Nick Coghlan promised to try to post his `urischemes
module`_ (a derivative of Paul's `uriparse module`_) to the `Python
Package Index`_.

.. _earlier discussions about replacing urlparse:
http://www.python.org/dev/summary/2006-06-01_2006-06-15/#rfc-3986-uniform-resource-identifiers-uris
.. _uriparse module: http://bugs.python.org/1462525
.. _urischemes module: http://bugs.python.org/1500504
.. _Python Package Index: http://www.python.org/pypi

Contributing threads:

- `patch 1462525 or similar solution?
`__
- `Path object design
`__

--
Importing .py, .pyc and .pyo files
--

Martin v. Löwis brought up `Osvaldo Santana's patch`_ which would have
made Python search for both .pyc and .pyo files regardless of whether
or not the optimize flag, "-OO", was set (like zipimporter does).
Without this patch, when "-OO" was given, Python never looked for .pyc
files. Some people thought that an extra ``stat()`` call or directory
listing to check for the other file would be too expensive, but no one
profiled the various versions of the code so the cost was unclear.
People were leaning towards removing the extra functionalit