Re: [Python-Dev] Path object design
Steve Holden wrote: > Greg Ewing wrote: >> Mike Orr wrote: >> Having said that, I can see there could be an >> element of confusion in calling it "join". >> > Good point. "relativise" might be appropriate, though something shorter > would be better. > > regards > Steve The term used in many languages for this sort of operation is "combine". (See .Net System.IO.Path for an example.) I kind of like the term - it implies that you are mixing two paths together, but it doesn't imply that the combination will be additive. - Talin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Steve Holden wrote: > Greg Ewing wrote: > >>Having said that, I can see there could be an >>element of confusion in calling it "join". >> > > Good point. "relativise" might be appropriate, Sounds like something to make my computer go at warp speed, which would be nice, but I won't be expecting a patch any time soon. :-) -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] idea for data-type (data-format) PEP
On Nov 2, 2006, at 9:25 PM, Greg Ewing wrote: > > I think what would happen if you were interoperating with > ctypes is that you would get a datatype describing one > element of the array, together with the shape information, > and construct a ctypes array type from that. And going > the other way, from a ctypes array type you would extract > an element datatype and a shape. Correct, assuming Travis' approach is accepted. However I understood that Martin was suggesting that ctypes types should be used to describe the structure of the buffer. Thus a buffer containing 10 integers would report its datatype as c_int*10. I was probably mistaken and Martin was suggesting the same as you. In this case extended buffer protocol would still use a different model from ctype and "don't reinvent the wheel" argument goes away. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Adding data-type objects to Python
Paul Moore gmail.com> writes:
> Somewhat. My understanding is that the python-level buffer object is
> frowned upon as not good practice, and is scheduled for removal at
> some point (Py3K, quite possibly?) Hence, any code that uses buffer()
> feels like it "needs" to be replaced by something "more acceptable".
Python 2.x buffer object serves two distinct purposes. First, it is a
"mutable string" object and this is definitely not going away being
replaced by the bytes object. (Interestingly, this functionality is not
exposed to python, but C extension modules can call
PyBuffer_New(size) to create a buffer.) Second, it is a "view" into any
object supporting buffer protocol. For a while this usage was indeed
frowned upon because buffer objects held the pointer obtained from
bf_get*buffer for too long causing memory errors in situations like
this:
>>> a = array('c', "x"*10)
>>> b = buffer(a, 5, 2)
>>> a.extend('x'*1000)
>>> str(b)
'xx'
This problem was fixed more than two years ago.
--
r35400 | nascheme | 2004-03-10
Make buffer objects based on mutable objects (like array) safe.
--
Even though it was suggested in the past that buffer *object*
should be deprecated as unsafe, I don't remember seeing a call
to deprecate the buffer protocol.
> So although I understand the use you suggest, it's not compelling to
> me because I am left with the feeling that I wish I knew "the way to
> do it that didn't need the buffer object" (even though I realise
> intellectually that such a way may not exist).
>
As I explained in another post, I used buffer object as an example of
an object that supports buffer protocol, but does not export type
information in the form usable by numpy.
Here is another way to illustrate the problem:
>>> a = numpy.array(array.array('H', [1,2,3]))
>>> b = numpy.array([1,2,3],dtype='H')
>>> a.dtype == b.dtype
False
With the extended buffer protocol it will be possible for numpy.array(..)
to realize that array.array('H', [1,2,3]) is a sequence of unsigned short
integers and convert it accordingly. Currently numpy has to go through
the sequence protocol to create a numpy.array from an array.array and
loose the type information.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Greg Ewing wrote:
> Mike Orr wrote:
>
>
>>>* This is confusing as heck:
>>> >>> os.path.join("hello", "/world")
>>> '/world'
>
>
> It's only confusing if you're not thinking of
> pathnames as abstract entities.
>
> There's a reason for this behaviour -- it's
> so you can do things like
>
>full_path = os.path.join(default_dir, filename_from_user)
>
> where filename_from_user can be either a relative
> or absolute path at his discretion.
>
> In other words, os.path.join doesn't just mean "join
> these two paths together", it means "interpret the
> second path in the context of the first".
>
> Having said that, I can see there could be an
> element of confusion in calling it "join".
>
Good point. "relativise" might be appropriate, though something shorter
would be better.
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] idea for data-type (data-format) PEP
Alexander Belopolsky wrote: > My main concern was that in ctypes the size of an array is a part of > the datatype object and this seems to be redundant if used for the > buffer protocol. Buffer protocol already reports the size of the > buffer as a return value of bf_get*buffer methods. I think what would happen if you were interoperating with ctypes is that you would get a datatype describing one element of the array, together with the shape information, and construct a ctypes array type from that. And going the other way, from a ctypes array type you would extract an element datatype and a shape. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] idea for data-type (data-format) PEP
On Nov 2, 2006, at 8:04 PM, Greg Ewing wrote:
>
> I think the datatype object should almost certainly
> be immutable. Since it's separated from the data
> it's describing, it's possible for one datatype
> object to describe multiple chunks of data. So
> you wouldn't want to mutate one in case it's being
> used for something else that you don't know about.
I only mentioned that the datatype object would be mutable at C level
because changing the object instead of deleting and creating a new
one could be a valid optimization in situations where the object is
know not to be shared.
My main concern was that in ctypes the size of an array is a part of
the datatype object and this seems to be redundant if used for the
buffer protocol. Buffer protocol already reports the size of the
buffer as a return value of bf_get*buffer methods.
In another post, Greg Ewing wrote:
> > numpy.array(array.array('d',[1,2,3]))
> >
> > and leave-out the buffer object all together.
> I think the buffer object in his example was just a
> placeholder for "some arbitrary object that supports
> the buffer interface", not necessarily another NumPy
> array.
Yes, thanks. In fact numpy.array(array.array('d',[1,2,3])) already
works in numpy (I think because numpy knows about the standard
library array type). In my example, I wanted to use an object that
supports buffer protocol and little else.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
On 01:04 am, [EMAIL PROTECTED] wrote:>[EMAIL PROTECTED] wrote:>If you're serious about writing platform-agnostic>pathname code, you don't put slashes in the arguments>at all. Instead you do>> os.path.join("hello", "slash", "world")>>Many of the other things you mention are also a>result of not treating pathnames as properly opaque>objects.Of course nobody who cares about these issues is going to put constant forward slashes into pathnames. The point is not that you'll forget you're supposed to be dealing with pathnames; the point is that you're going to get input from some source that you've got very little control over, and *especially* if that source is untrusted (although sometimes just due to mistakes) there are all kinds of ways it can trip you up. Did you accidentally pass it through something that doubles or undoubles all backslashes, etc. Sometimes these will result in harmless errors anyway, sometimes it's a critical error that will end up trying to delete /usr instead of /home/user/installer-build/ROOT/usr. If you have the path library catching these problems for you then a far greater percentage fall into the former category.>If you're saying that the fact they're strings makes>it easy to forget that you're supposed to be treating>them opaquely,That's exactly what I'm saying.>> * although individual operations are atomic, shutil.copytree and friends >>aren't. I've often seen python programs confused by partially-copied trees >>of files.>I can't see how this can be even remotely regarded>as a pathname issue, or even a filesystem interface>issue. It's no different to any other situation>where a piece of code can fall over and leave a>partial result behind.It is a bit of a stretch, I'll admit, but I included it because it is a weakness of the path library that it is difficult to do the kind of parallel iteration required to implement tree-copying yourself. If that were trivial, then you could write your own file-copying loop and cope with errors yourself.___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Adding data-type objects to Python
Travis E. Oliphant wrote: > We have T_UBYTE and T_BYTE, etc. defined > in structmember.h already. Should we just re-use those #defines while > adding to them to make an easy to use interface for primitive types? They're mixed up with size information, though, which we don't want to do. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
[EMAIL PROTECTED] wrote: > Relative > paths, if they should exist at all, should have to be explicitly linked > as relative to something *else* (e.g. made absolute) before they can be > used. If paths were opaque objects, this could be enforced by not having any way of constructing a path that wasn't rooted in some existing absolute path. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Mike Orr wrote:
> I have no idea why Microsoft thought it was a good idea to
> put the seven-odd device files in every directory. Why not force
> people to type the colon ("CON:").
Yes, this is a particularly stupid piece of braindamage
on the part of the designers of MS-DOS. As far as I
remember, even CP/M (which was itself a severely
warped and twisted version of RT11) had the good
sense to put colons on the end of such things.
But maybe "design" is too strong a word to apply
to MS-DOS...
Anyhow, I think I agree that there's really nothing
a path library can do about this. Whatever it tries
to do, the fact will remain that it's impossible to
have a regular file called "con", and users will
have to live with that.
--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
[EMAIL PROTECTED] wrote:
>>>> os.path.join("hello", "slash/world")
>'hello/slash/world'
>>>> os.path.join("hello", "slash//world")
>'hello/slash//world'
>Trying to formulate a general rule for what the arguments to
> os.path.join are supposed to be is really hard.
If you're serious about writing platform-agnostic
pathname code, you don't put slashes in the arguments
at all. Instead you do
os.path.join("hello", "slash", "world")
Many of the other things you mention are also a
result of not treating pathnames as properly opaque
objects.
If you're saying that the fact they're strings makes
it easy to forget that you're supposed to be treating
them opaquely, there may be merit in that view. It
would be an argument for making path objects a
truly opaque type instead of a subclass of string
or tuple.
> * although individual operations are atomic, shutil.copytree and
> friends aren't. I've often seen python programs confused by
> partially-copied trees of files.
I can't see how this can be even remotely regarded
as a pathname issue, or even a filesystem interface
issue. It's no different to any other situation
where a piece of code can fall over and leave a
partial result behind. As always, the cure is
defensive coding (clean up a partial result on error,
or be prepared to tolerate the presence of a previous
partial result when re-trying).
It could be argued that shutil.copytree should clean
up after itself if there is an error, but that might
not be what you want -- e.g. you might want to find
out how far it got, and maybe carry on from there
next time. It's probably better to leave things like
that to the caller.
--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] idea for data-type (data-format) PEP
Alexander Belopolsky wrote: > That's a question for Travis, but I would think that they would be > immutable at the Python level, but mutable at the C level. Well, anything's mutable at the C level -- the question is whether you *should* be mutating it. I think the datatype object should almost certainly be immutable. Since it's separated from the data it's describing, it's possible for one datatype object to describe multiple chunks of data. So you wouldn't want to mutate one in case it's being used for something else that you don't know about. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Adding data-type objects to Python
Travis Oliphant wrote:
> or just
>
> numpy.array(array.array('d',[1,2,3]))
>
> and leave-out the buffer object all together.
I think the buffer object in his example was just a
placeholder for "some arbitrary object that supports
the buffer interface", not necessarily another NumPy
array.
--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Mike Orr wrote:
>> * This is confusing as heck:
>> >>> os.path.join("hello", "/world")
>> '/world'
It's only confusing if you're not thinking of
pathnames as abstract entities.
There's a reason for this behaviour -- it's
so you can do things like
full_path = os.path.join(default_dir, filename_from_user)
where filename_from_user can be either a relative
or absolute path at his discretion.
In other words, os.path.join doesn't just mean "join
these two paths together", it means "interpret the
second path in the context of the first".
Having said that, I can see there could be an
element of confusion in calling it "join".
--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Extending the buffer protocol to share array information.
Fredrik Lundh wrote: > the "right solution" for things like this is an *API* that lets you do > things like: > > view = object.acquire_view(region, supported formats) And how do you describe the "supported formats"? That's where Travis's proposal comes in, as far as I can see. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] idea for data-type (data-format) PEP
Alexander Belopolsky wrote: > In ctypes arrays of different > shapes are represented using different types. As a result, if the object > exporting its buffer is resized, the datatype object cannot be reused, it > has to be replaced. I was thinking about that myself the other day. I was thinking that both ctypes and NumPy arrays + proposed_type_descriptor provide a way of describing an array of binary data and providing Python-level access to that data. So a NumPy array and an instance of a ctypes type that happens to describe an array are very similar things. I was wondering whether they could be unified somehow. But then I realised that the ctypes array is a fixed-size array, whereas NumPy's notion of an array is rather more flexible. So they're not really the same thing after all. However, the *elements* of the array are fixed size in both cases, so the respective descriptions of the element type could potentially have something in common. My current take on the situation is that Travis is probably right about ctypes types being too cumbersome for what he has in mind. The next best thing would be to make them interoperate: have an easy way of getting a ctypes type corresponding to a given data layout description and vice versa. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] idea for data-type (data-format) PEP
T > > IIUC, so far the 'data-object' carries information about the structure > of the data it describes. > > Couldn't it go a step further and have also some functionality? > Converting the data into a Python object and back? > Yes, I had considered it to do that. That's why the setfunc and getfunc functions were written the way they were. -teo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Adding data-type objects to Python
On Nov 2, 2006, at 9:35 PM, Thomas Heller wrote: Ronald Oussoren schrieb: On Oct 31, 2006, at 6:38 PM, Thomas Heller wrote: This mechanism is probably a hack because it'n not possible to add C accessible fields to type objects, on the other hand it is extensible (in principle, at least). I better start rewriting PyObjC then :-). PyObjC stores some addition information in the type objects that are used to describe Objective-C classes (such as a reference to the proxied class). IIRC This has been possible from Python 2.3. I assume you are referring to the code in pyobjc/Modules/objc/objc- class.h Yes. If this really is reliable I should better start rewriting ctypes then ;-). Hm, I always thought there was some additional magic going on with type objects, fields appended dynamically at the end or whatever. There is such magic, but that magic was updated in Python 2.3 to allow type-object extensions like this. Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Adding data-type objects to Python
Ronald Oussoren schrieb: > On Oct 31, 2006, at 6:38 PM, Thomas Heller wrote: > >> >> This mechanism is probably a hack because it'n not possible to add >> C accessible >> fields to type objects, on the other hand it is extensible (in >> principle, at least). > > I better start rewriting PyObjC then :-). PyObjC stores some addition > information in the type objects that are used to describe Objective-C > classes (such as a reference to the proxied class). > > IIRC This has been possible from Python 2.3. I assume you are referring to the code in pyobjc/Modules/objc/objc-class.h ? If this really is reliable I should better start rewriting ctypes then ;-). Hm, I always thought there was some additional magic going on with type objects, fields appended dynamically at the end or whatever. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] idea for data-type (data-format) PEP
Travis E. Oliphant schrieb: > Travis E. Oliphant wrote: >> Thanks for all the comments that have been given on the data-type >> (data-format) PEP. I'd like opinions on an idea for revising the PEP I >> have. > >> >> 1) We could define a special string-syntax (or list syntax) that covers >> every special case. The array interface specification goes this >> direction and it requires no new Python types. This could also be seen >> as an extension of the "struct" module to allow for nested structures, etc. >> >> 2) We could define a Python object that specifically carries data-format >> information. >> >> >> Does that explain the goal of what I'm trying to do better? > > In other-words, what I'm saying is I really want a PEP that does this. > Could we have a discussion about what the best way to communicate > data-format information across multiple extension modules would look > like. I'm not saying my (pre-)PEP is best. The point of putting it in > it's infant state out there is to get the discussion rolling, not to > claim I've got all the answers. IIUC, so far the 'data-object' carries information about the structure of the data it describes. Couldn't it go a step further and have also some functionality? Converting the data into a Python object and back? This is what the ctypes SETFUNC and GETFUNC functions do, and what also is implemented in the struct module... Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Adding data-type objects to Python
Martin v. Löwis wrote: > Travis E. Oliphant schrieb: > >>>2. Should primitive type codes be characters or integers (from an enum) at >>>C level? >>>- I prefer integers >> >>>3. Should size be expressed in bits or bytes? >>>- I prefer bits >>> >> >>So, you want an integer enum for the "kind" and an integer for the >>bitsize? That's fine with me. >> >>One thing I just remembered. We have T_UBYTE and T_BYTE, etc. defined >>in structmember.h already. Should we just re-use those #defines while >>adding to them to make an easy to use interface for primitive types? > > > Notice that those type codes imply sizes, namely the platform sizes > (where "platform" always means "what the C compiler does"). So if > you want to have platform-independent codes as well, you shouldn't > use the T_ codes. > In NumPy we've found it convenient to use both. Basically, we've set up a header file that "does the translation" using #defines and typedefs to create things like (on a 32-bit platform) typedef npy_int32 int #define NPY_INT32 NPY_INT So, that either the T_code-like enum or the bit-width can be used interchangable. Typically people want to specify bit-widths (and see their data-types in bit-widths) but in C-code that implements something you need to use one of the platform integers. I don't know if we really need to bring all of that over. -Travis ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Adding data-type objects to Python
On 11/2/06, Travis Oliphant <[EMAIL PROTECTED]> wrote: > What do you mean by "manipulate the data." The proposal for a > data-format object would help you describe that data in a standard way > and therefore share that data between several library that would be able > to understand the data (because they all use and/or understand the > default Python way to handle data-formats). > > It would be up to the other packages to "manipulate" the data. Yes, some other messages I read since I posted this clarified it for me. Essentially, as a Python programmer, there's nothing in the PEP for me - it's for extension writers (and maybe writers of some lower-level Python modules? I'm not sure about this). So as I'm not really the target audience, I won't comment further. > So, what you would be able to do is take your byte-string and create a > buffer object which you could then share with other packages: > > Example: > > b = buffer(bytestr, format=data_format_object) > > Now. > > a = numpy.frombuffer(b) > a['field1'] # prints data stored in the field named "field1" > > etc. > > Or. > > cobj = ctypes.frombuffer(b) > > # Now, cobj is a ctypes object that is basically a "structure" that can > be passed # directly to your C-code. > > Does this help? Somewhat. My understanding is that the python-level buffer object is frowned upon as not good practice, and is scheduled for removal at some point (Py3K, quite possibly?) Hence, any code that uses buffer() feels like it "needs" to be replaced by something "more acceptable". So although I understand the use you suggest, it's not compelling to me because I am left with the feeling that I wish I knew "the way to do it that didn't need the buffer object" (even though I realise intellectually that such a way may not exist). Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Mini Path object
On 11/2/06, Mike Orr <[EMAIL PROTECTED]> wrote: > Given the widely-diverging views on what, if anything, should be done > to os.path, how about we make a PEP and a standalone implementation of > (1) for now, and leave (2) and everything else for a later PEP. Why write a PEP at this stage? Just release your proposal as a module, and see if people use it. If they do, write a PEP to include it in the stdlib. (That's basically what happened with the original PEP - it started off proposing Jason Orendorff's path module IIRC). >From what you're proposing, I may well use such a module, if it helps :-) (But I'm not sure I'd vote for it in to go the stdlib without having survived as an external module first) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
