Re: [Python-Dev] PATCH/RFC for AF_NETLINK support

["Martin v. Löwis"]

Philippe Biondi wrote:
I've done a small patch to use linux AF_NETLINK sockets (see below).
Please comment!
I have a high-level comment - python-dev is normally the wrong place
for patches; please submit them to sf.net/projects/python instead.
Apart from that, the patch looks fine.
Is there a reason for recvmsg() and sendmsg() not to be implemented
yet in socketmodule ?
I'm not sure what you mean by "implemented": these functions are
implemented by the operating system, not in the socketmodule.
If you ask "why are they not exposed to Python yet?": There has been no
need to do so, so far. What do I get with recvmsg that I cannot get with
recv/recvfrom just as well?
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PATCH/RFC for AF_NETLINK support

[David Wilson]

On Mon, Jan 10, 2005 at 05:17:49PM +0100, Philippe Biondi wrote:

> I've done a small patch to use linux AF_NETLINK sockets (see below).
> Please comment!

As of 2.6.10, a very useful new netlink family was merged -
NETLINK_KOBJECT_UEVENT. I'd imagine quite a lot of interest from Python
developers for NETLINK support will come from this new interface in the
coming years.

http://lwn.net/Articles/101210/
http://lkml.org/lkml/2004/9/10/315
http://vrfy.org/projects/kevents/
http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.10

I would like to see (optional?) support for this before your patch is
merged. I have a long-term interest in a Python-based service control /
init replacement / system management application, for use in specialised
environments. I could definately use this. :)

Thanks,


David.

-- 
Harmless - and in its harmlessness, diabolical.
-- The Mold Of Yancy (Philip K. Dick)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: [Csv] csv module TODO list

[Neal Norwitz]

On Wed, Jan 05, 2005 at 10:08:49PM +1100, Andrew McNamara wrote:
> >Also, review comments from Neal Norwitz, 22 Mar 2003 (some of these should
> >already have been addressed):
> 
> I should apologise to Neal here for not replying to him at the time.

Hey, I'm impressed you got to them.  :-) I completely forgot about it.

> >* rather than use PyErr_BadArgument, should you use assert?
> >(first example, Dialect_set_quoting, line 218)
> 
> You mean C assert()? I don't think I'm really following you here -
> where would the type of the object be checked in a way the user could
> recover from?

IIRC, I meant C assert().  This goes back to a discussion a long time
ago about what is the preferred way to handle invalid arguments.
I doubt it's important to change.

> >* I think you need PyErr_NoMemory() before returning on line 768, 1178
> 
> The examples I looked at in the Python core didn't do this - are you sure?
> (now lines 832 and 1280). 

Originally, they were a plain PyObject_NEW().  Now they are a
PyObject_GC_New() so it seems no further change is necessary.

> >* is PyString_AsString(self->dialect->lineterminator) on line 994
> >guaranteed not to return NULL?  If not, it could crash by
> >passing to memmove.
> >* PyString_AsString() can return NULL on line 1048 and 1063, 
> >the result is passed to join_append()
> 
> Looking at the PyString_AsString implementation, it looks safe (we ensure
> it's really a string elsewhere)?

Ok.  Then it should be fine.  I spot checked lineterminator and it
looked ok.

> >* iteratable should be iterable?  (line 1088)
> 
> Sorry, I don't know what you're getting at here? (now line 1162).

Heh, I had to read that twice myself.  It was a typo (assuming
I wasn't completely wrong)--an extra "at", but it doesn't exist
any longer.

I don't think there are any changes remaining to be done from my
original code review.

BTW, I always try to run valgrind before a release, especially
major releases.

Neal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PATCH/RFC for AF_NETLINK support

[Philippe Biondi]

Hi,

I've done a small patch to use linux AF_NETLINK sockets (see below).
Please comment!

Is there a reason for recvmsg() and sendmsg() not to be implemented
yet in socketmodule ?


The integration with autoconf has not been done, even if
this patch should be ok :

--- configure.in.ori2005-01-10 17:09:32.0 +0100
+++ configure.in2005-01-06 18:53:18.0 +0100
@@ -967,7 +967,7 @@
 sys/audioio.h sys/bsdtty.h sys/file.h sys/loadavg.h sys/lock.h sys/mkdev.h \
 sys/modem.h \
 sys/param.h sys/poll.h sys/select.h sys/socket.h sys/time.h sys/times.h \
-sys/un.h sys/utsname.h sys/wait.h pty.h libutil.h \
+sys/un.h linux/netlink.h sys/utsname.h sys/wait.h pty.h libutil.h \
 sys/resource.h netpacket/packet.h sysexits.h bluetooth.h \
 bluetooth/bluetooth.h)
 AC_HEADER_DIRENT
--- pyconfig.h.ori  2005-01-10 17:11:11.0 +0100
+++ pyconfig.h  2005-01-06 19:27:33.0 +0100
@@ -559,6 +559,9 @@
 /* Define to 1 if you have the  header file. */
 #define HAVE_SYS_UN_H 1

+/* Define to 1 if you have the  header file. */
+#define HAVE_LINUX_NETLINK_H 1
+
 /* Define to 1 if you have the  header file. */
 #define HAVE_SYS_UTSNAME_H 1



--- socketmodule.h.ori  2005-01-07 19:25:18.0 +0100
+++ socketmodule.h  2005-01-06 18:20:54.0 +0100
@@ -32,6 +32,12 @@
 # undef AF_UNIX
 #endif

+#ifdef HAVE_LINUX_NETLINK_H
+# include 
+#else
+#  undef AF_NETLINK
+#endif
+
 #ifdef HAVE_BLUETOOTH_BLUETOOTH_H
 #include 
 #include 
@@ -87,6 +93,9 @@ typedef struct {
 #ifdef AF_UNIX
struct sockaddr_un un;
 #endif
+#ifdef AF_NETLINK
+   struct sockaddr_nl nl;
+#endif
 #ifdef ENABLE_IPV6
struct sockaddr_in6 in6;
struct sockaddr_storage storage;
--- socketmodule.c.ori  2005-01-07 19:25:19.0 +0100
+++ socketmodule.c  2005-01-10 17:04:38.0 +0100
@@ -948,6 +948,14 @@ makesockaddr(int sockfd, struct sockaddr
}
 #endif /* AF_UNIX */

+#if defined(AF_NETLINK)
+   case AF_NETLINK:
+   {
+   struct sockaddr_nl *a = (struct sockaddr_nl *) addr;
+   return Py_BuildValue("ii", a->nl_pid, a->nl_groups);
+   }
+#endif /* AF_NETLINK */
+
 #ifdef ENABLE_IPV6
case AF_INET6:
{
@@ -1084,6 +1092,31 @@ getsockaddrarg(PySocketSockObject *s, Py
}
 #endif /* AF_UNIX */

+#if defined(AF_NETLINK)
+   case AF_NETLINK:
+   {
+   struct sockaddr_nl* addr;
+   int pid, groups;
+   addr = (struct sockaddr_nl *)&(s->sock_addr).nl;
+   if (!PyTuple_Check(args)) {
+   PyErr_Format(
+   PyExc_TypeError,
+   "getsockaddrarg: "
+   "AF_NETLINK address must be tuple, not %.500s",
+   args->ob_type->tp_name);
+   return 0;
+   }
+   if (!PyArg_ParseTuple(args, "II", &pid, &groups))
+   return 0;
+   addr->nl_family = AF_NETLINK;
+   addr->nl_pid = pid;
+   addr->nl_groups = groups;
+   *addr_ret = (struct sockaddr *) addr;
+   *len_ret = sizeof(*addr);
+   return 1;
+   }
+#endif
+
case AF_INET:
{
struct sockaddr_in* addr;
@@ -1280,6 +1313,13 @@ getsockaddrlen(PySocketSockObject *s, so
return 1;
}
 #endif /* AF_UNIX */
+#if defined(AF_NETLINK)
+   case AF_NETLINK:
+   {
+   *len_ret = sizeof (struct sockaddr_nl);
+   return 1;
+   }
+#endif

case AF_INET:
{
@@ -3938,8 +3978,20 @@ init_socket(void)
PyModule_AddIntConstant(m, "AF_KEY", AF_KEY);
 #endif
 #ifdef AF_NETLINK
-   /*  */
+   /* Netlink socket */
PyModule_AddIntConstant(m, "AF_NETLINK", AF_NETLINK);
+   PyModule_AddIntConstant(m, "NETLINK_ROUTE", NETLINK_ROUTE);
+   PyModule_AddIntConstant(m, "NETLINK_SKIP", NETLINK_SKIP);
+   PyModule_AddIntConstant(m, "NETLINK_USERSOCK", NETLINK_USERSOCK);
+   PyModule_AddIntConstant(m, "NETLINK_FIREWALL", NETLINK_FIREWALL);
+   PyModule_AddIntConstant(m, "NETLINK_TCPDIAG", NETLINK_TCPDIAG);
+   PyModule_AddIntConstant(m, "NETLINK_NFLOG", NETLINK_NFLOG);
+   PyModule_AddIntConstant(m, "NETLINK_XFRM", NETLINK_XFRM);
+   PyModule_AddIntConstant(m, "NETLINK_ARPD", NETLINK_ARPD);
+   PyModule_AddIntConstant(m, "NETLINK_ROUTE6", NETLINK_ROUTE6);
+   PyModule_AddIntConstant(m, "NETLINK_IP6_FW", NETLINK_IP6_FW);
+   PyModule_AddIntConstant(m, "NETLINK_DNRTMSG", NETLINK_DNRTMSG);
+   PyModule_AddIntConstant(m, "NETLINK_TAPBASE", NETLINK_TAPBASE);
 #endif
 #ifdef AF_ROUTE
/* Alias to emulate 4.4BSD */


-- 
Philippe Biondi   SecDev.org
Security Consultant/R&D http://www.secdev.org
PGP KeyID:3D9A43E2  FingerPrint:C40A772533730E39330DC0985EE8FF5F3D9A43E2

___

Re: [Python-Dev] PEP 246, redux

[Phillip J. Eby]

At 05:42 PM 1/10/05 -0500, Bob Ippolito wrote:
On Jan 10, 2005, at 16:38, Phillip J. Eby wrote:
At 07:42 PM 1/10/05 +0100, Alex Martelli wrote:
On 2005 Jan 10, at 18:43, Phillip J. Eby wrote:
   ...
I am not saying we shouldn't have a tp_conform; just suggesting that it 
may be appropriate for functions and modules (as well as classic 
classes) to have their tp_conform delegate back to 
self.__dict__['__conform__'] instead of a null implementation.
I have not considered conformance of such objects as functions or 
modules; if that is important,
It's used in at least Zope and PEAK; I don't know if it's in use in Twisted.
SVN trunk of Twisted (what will be 2.0) uses zope.interface.
What I meant was, I don't know if Twisted actually *uses* interface 
declarations for modules and functions.  It has the ability to do so, 
certainly.  I was just saying I didn't know if the ability is actually used.

PEAK uses some interfaces for functions, but I don't think I've ever used 
them for modules, and can think of only one place in PEAK where it would 
make sense to declare a module as supporting an interface.  Zope policy is 
to use interfaces for *everything*, though, including documenting the 
interface provided by modules.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Bob Ippolito]

On Jan 10, 2005, at 16:38, Phillip J. Eby wrote:
At 07:42 PM 1/10/05 +0100, Alex Martelli wrote:
On 2005 Jan 10, at 18:43, Phillip J. Eby wrote:
   ...
I am not saying we shouldn't have a tp_conform; just suggesting that 
it may be appropriate for functions and modules (as well as classic 
classes) to have their tp_conform delegate back to 
self.__dict__['__conform__'] instead of a null implementation.
I have not considered conformance of such objects as functions or 
modules; if that is important,
It's used in at least Zope and PEAK; I don't know if it's in use in 
Twisted.
SVN trunk of Twisted (what will be 2.0) uses zope.interface.  It still 
has the older stuff implemented as a wrapper on top of zope.interface, 
but I think the guideline is to just use zope.interface directly for 
new code dependent on Twisted 2.0.

-bob
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Phillip J. Eby]

At 07:42 PM 1/10/05 +0100, Alex Martelli wrote:
On 2005 Jan 10, at 18:43, Phillip J. Eby wrote:
   ...
At 03:42 PM 1/10/05 +0100, Alex Martelli wrote:
The fourth case above is subtle.  A break of substitutability can
occur when a subclass changes a method's signature, or restricts
the domains accepted for a method's argument ("co-variance" on
arguments types), or extends the co-domain to include return
values which the base class may never produce ("contra-variance"
on return types).  While compliance based on class inheritance
_should_ be automatic, this proposal allows an object to signal
that it is not compliant with a base class protocol.
-1 if this introduces a performance penalty to a wide range of 
adaptations (i.e. those using abstract base classes), just to support 
people who want to create deliberate Liskov violations.  I personally 
don't think that we should pander to Liskov violators, especially since 
Guido seems to be saying that there will be some kind of interface 
objects available in future Pythons.
If interfaces can ensure against Liskov violations in instances of their 
subclasses, then they can follow the "case (a)" fast path, sure.
Inheriting from an interface (in Guido's current proposal, as per his 
Artima blog) is a serious commitment from the inheritor's part; inheriting 
from an ordinary type, in real-world current practice, need not be -- too 
many cases of assumed covariance, for example, are around in the wild, to 
leave NO recourse in such cases and just assume compliance.
I understand that, sure.  But I don't understand why we should add 
complexity to PEP 246 to support not one but *two* bad practices: 1) 
implementing Liskov violations and 2) adapting to concrete classes.  It is 
only if you are doing *both* of these that this extra feature is needed.

If it were to support some kind of backward compatibility, that would be 
understandable.  However, in practice, I don't know of anybody using 
adapt(x,ConcreteClass), and even if they did, the person subclassing 
ConcreteClass will need to change their subclass to raise LiskovViolation, 
so why not just switch to delegation?

Anyway, it seems to me a bad idea to add complexity to support this 
case.  Do you have a more specific example of a situation in which a Liskov 
violation coupled to concrete class adaptation is a good idea?  Or am I 
missing something here?


I am not saying we shouldn't have a tp_conform; just suggesting that it 
may be appropriate for functions and modules (as well as classic classes) 
to have their tp_conform delegate back to self.__dict__['__conform__'] 
instead of a null implementation.
I have not considered conformance of such objects as functions or modules; 
if that is important,
It's used in at least Zope and PEAK; I don't know if it's in use in Twisted.

 I need to add it to the reference implementation in the PEP.  I'm 
reluctant to just get __conform__ from the object, though; it leads to 
all sort of issues with a *class* conforming vs its *instances*, 
etc.  Maybe Guido can Pronounce a little on this sub-issue...
Actually, if you looked at the field-tested implementations of the old PEP 
246, they actually have code that deals with this issue effectively, by 
recognizing TypeError when raised by attempting to invoke __adapt__ or 
__conform__ with the wrong number of arguments or argument types.  (The 
traceback for such errors does not include a frame for the called method, 
versus a TypeError raised *within* the function, which does have such a 
frame.  AFAIK, this technique should be compatible with any Python 
implementation that has traceback objects and does signature validation in 
its "native" code rather than in a new Python frame.)


I don't see the benefit of LiskovViolation, or of doing the exact type 
check vs. the loose check.  What is the use case for these?  Is it to 
allow subclasses to say, "Hey I'm not my superclass?"  It's also a bit 
confusing to say that if the routines "raise any other exceptions" 
they're propagated.  Are you saying that LiskovViolation is *not* propagated?
Indeed I am -- I thought that was very clearly expressed!
The PEP just said that it would be raised by __conform__ or __adapt__, not 
that it would be caught by adapt() or that it would be used to control the 
behavior in that way.  Re-reading, I see that you do mention it much 
farther down.  But at the point where __conform__ and __adapt__ are 
explained, it has not been explained that adapt() should catch the error or 
do anything special with it.  It is simply implied by the "to prevent this 
default behavior" at the end of the section.  If this approach is accepted, 
the description should be made explicit, becausse for me at least it 
required a retroactive re-interpretation of the earlier part of the spec.


The previous version treated TypeError specially, but I think (on the 
basis of just playing around a bit, admittedly) that offers no real added 
va

Re: [Python-Dev] PEP 246, redux

[Phillip J. Eby]

At 02:27 PM 1/10/05 -0500, Clark C. Evans wrote:
If this proposal was "packaged" with an "interface" mechanism, would
this address your concern?  In this scenerio, there are two cases:
  - Older classes will most likely not have a __conform__ method.
  - Newer classes will use the 'interface' mechanism.
In this scenerio, there isn't a performance penalty for the
usual case; and for migration purposes, a flag could be added
to disable the checking.
As I said, after more thought, I'm actually less concerned about the 
performance than I am about even remotely encouraging the combination of 
Liskov violation *and* concrete adaptation targets.  But, if "after the 
dust settles" it turns out this is going to be supported after all, then we 
can worry about the performance if need be.

Note, however, that your statements actually support the idea of *not* 
adding a special case for Liskov violators.  If newer code uses interfaces, 
the Liskov-violation mechanism is useless.  If older code doesn't have 
__conform__, it cannot possibly *use* the Liskov-violation mechanism.

So, if neither old code nor new code will use the mechanism, why have it?  :)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Thomas Heller]

Alex Martelli <[EMAIL PROTECTED]> writes:

> PEP: 246
> Title: Object Adaptation

Minor nit (or not?): You could provide a pointer to the Liskov
substitution principle, for those reader that aren't too familiar with
that term.

Besides, the text mentions three times that LiskovViolation is a
subclass of AdaptionError (plus once in the ref impl section).

Thomas

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Phillip J. Eby]

At 04:16 PM 1/10/05 -0800, Michel Pelletier wrote:
> From: Guido van Rossum <[EMAIL PROTECTED]>
> Subject: Re: [Python-Dev] PEP 246, redux
>
> I'm wondering if someone could do a similar thing for PEP 245,
> interfaces syntax? Alex hinted that it's a couple of rounds behind the
> developments in Zope and Twisted.
Nothing implements 245, which is just about the syntax,
The comment Guido's alluding to was mine; I was referring to PEP 245's use 
of '__implements__', and the difference between what a "class implements" 
and an "instance provides".  Twisted and Zope's early implementations just 
looked for ob.__implements__, which leads to issues with distinguishing 
between what a "class provides" from what its "instances provide".

So, I was specifically saying that this aspect of PEP 245 (and Guido's 
basing a Python interface implementation thereon) should be re-examined in 
the light of current practices that avoid this issue.  (I don't actually 
know what Zope currently does; it was changed after I had moved to using 
PyProtocols.  But the PyProtocols test suite tests that Zope does in fact 
have correct behavior for instances versus classes, because it's needed to 
exercise the PyProtocols-Zope interop tests.)


I like implements, but any spelling works for me.  "implements" strikes me as
an elegant counterpart to "interface" and risks minimal breakage.  Can we
still import and say "implements()" for b/w compatibility and for those of us
who do want an explicit statement like that?
If I understand Guido's proposal correctly, it should be possible to make a 
backward-compatible 'implements()' declaration function.  Maybe not *easy*, 
but certainly possible.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: csv module TODO list

[Martin Bless]

On Mon, 10 Jan 2005 10:37:17 +1100, Andrew McNamara
<[EMAIL PROTECTED]> wrote:

>>csv.join(aList, e[, dialect='excel'[, fmtparam]]) -> str object

Oops, should have been

csv.join(aList [, dialect='excel'[, fmtparam]]) -> str object

>Yes, it's feasible,

Good!

>although newlines can be embedded in within fields
>of a CSV record, hence the use of the iterator, rather than working with
>strings.

In my use cases newlines usually don't come into play. It would be ok
for me if they were treated as any other char.

> In your example above, if the parser gets to the end of the
>string and finds it's still within a field, I'd propose just raising
>an exception.

Yes, that seems to be "the right answer".

>No promises, however - I only have a finite ammount of time to work on
>this at the moment.

Sure!

To my feeling these  "intelligent split and join" functions most
naturally should actually be string methods. I can see that -
considering the conceivable variety of dialects - this can't be done.
One more reason to have 'split' and 'join' available from the csv
module! 

mb - Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Michel Pelletier]

On Monday 10 January 2005 09:58 am, [EMAIL PROTECTED] wrote:

> Message: 3
> Date: Mon, 10 Jan 2005 07:46:39 -0800
> From: Guido van Rossum <[EMAIL PROTECTED]>
> Subject: Re: [Python-Dev] PEP 246, redux
> To: Alex Martelli <[EMAIL PROTECTED]>
> Cc: "Clark C.Evans" <[EMAIL PROTECTED]>, Python Dev
>   
> Message-ID: <[EMAIL PROTECTED]>
> Content-Type: text/plain; charset=US-ASCII
>
> > I had been promising to rewrite PEP 246 to incorporate the last several
> > years' worth of discussions &c about it, and Guido's recent "stop the
> > flames" artima blog post finally pushed me to complete the work.
> > Feedback is of course welcome, so I thought I had better repost it
> > here, rather than relying on would-be commenters to get it from CVS...
>
> Thanks for doing this, Alex! I yet have to read the whole thing [will
> attempt do so later today] but the few snippets I caught make me feel
> this is a big step forward.

Me too!  I didn't realize it the first time 246 came around how important 
adaptation was and how interfaces just aren't as useful without it.

>
> I'm wondering if someone could do a similar thing for PEP 245,
> interfaces syntax? Alex hinted that it's a couple of rounds behind the
> developments in Zope and Twisted. 

Nothing implements 245, which is just about the syntax, I intended to write 
another PEP describing an implementation, at the time Jim's original 
straw-man; which I'm glad I didn't do as it would have been a waste of time.  
Had I written that document, then it would be a copule of rounds behind Zope 
and Twisted.  But as it stands now nothing need be based on 245.

> I'm personally not keen on needing 
> *two* new keywords (interface and implements) so I hope that whoever
> does the rewrite could add a section on the advantages and
> disadvantages of the 'implements' keyword (my simplistic alternative
> proposal is to simply include interfaces in the list of bases in the
> class statement; the metaclass can then sort it out).

I like implements, but any spelling works for me.  "implements" strikes me as 
an elegant counterpart to "interface" and risks minimal breakage.  Can we 
still import and say "implements()" for b/w compatibility and for those of us 
who do want an explicit statement like that?

-Michel
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Clark C. Evans]

On Mon, Jan 10, 2005 at 01:34:59PM -0500, Phillip J. Eby wrote:
| The performance penalty I was talking about was for using an abstract 
| base class, in a subclass with a __conform__ method for conformance to 
| other protocols.  In this case, __conform__ will be uselessly called 
| every time the object is adapted to the abstract base class.

*nod*

If this proposal was "packaged" with an "interface" mechanism, would
this address your concern?  In this scenerio, there are two cases:

  - Older classes will most likely not have a __conform__ method.
  - Newer classes will use the 'interface' mechanism.

In this scenerio, there isn't a performance penalty for the 
usual case; and for migration purposes, a flag could be added
to disable the checking.

Best,

Clark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Michael Hudson]

Alex Martelli <[EMAIL PROTECTED]> writes:

> I didn't know about the "let the object lie" quirk in isinstance.  If
> that quirk is indeed an intended design feature, rather than an
> implementation 'oops', it might perhaps be worth documenting it more
> clearly; I do not find that clearly spelled out in the place I'd
> expect it to be, namely
>  under 'isinstance'.

Were you not at the PyPy sprint where bugs in some __getattr__ method
caused infinite recursions on the isinstance's code attempting to
access __class__?  The isinstance code then silently eats the error,
so we had (a) a massive slowdown and (b) isinstance failing in an
"impossible" way.  A clue was that if you ran the code on OS X with
its silly default stack limits the code dumped core instead of going
slowly insane.

This is on quirk I'm not likely to forget in a hurry...

Cheers,
mwh

-- 
  If trees could scream, would we be so cavalier about cutting them
  down? We might, if they screamed all the time, for no good reason.
-- Jack Handey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Alex Martelli]

On 2005 Jan 10, at 18:43, Phillip J. Eby wrote:
   ...
At 03:42 PM 1/10/05 +0100, Alex Martelli wrote:
The fourth case above is subtle.  A break of substitutability can
occur when a subclass changes a method's signature, or restricts
the domains accepted for a method's argument ("co-variance" on
arguments types), or extends the co-domain to include return
values which the base class may never produce ("contra-variance"
on return types).  While compliance based on class inheritance
_should_ be automatic, this proposal allows an object to signal
that it is not compliant with a base class protocol.
-1 if this introduces a performance penalty to a wide range of 
adaptations (i.e. those using abstract base classes), just to support 
people who want to create deliberate Liskov violations.  I personally 
don't think that we should pander to Liskov violators, especially 
since Guido seems to be saying that there will be some kind of 
interface objects available in future Pythons.
If interfaces can ensure against Liskov violations in instances of 
their subclasses, then they can follow the "case (a)" fast path, sure.  
Inheriting from an interface (in Guido's current proposal, as per his 
Artima blog) is a serious commitment from the inheritor's part; 
inheriting from an ordinary type, in real-world current practice, need 
not be -- too many cases of assumed covariance, for example, are around 
in the wild, to leave NO recourse in such cases and just assume 
compliance.


Just like any other special method in today's Python, __conform__
is meant to be taken from the object's class, not from the object
itself (for all objects, except instances of "classic classes" as
long as we must still support the latter).  This enables a
possible 'tp_conform' slot to be added to Python's type objects in
the future, if desired.
One note here: Zope and PEAK sometimes use interfaces that a function 
or module may implement.  PyProtocols' implementation does this by 
adding a __conform__ object to the function's dictionary so that the 
function can conform to a particular signature.  If and when 
__conform__ becomes tp_conform, this may not be necessary any more, at 
least for functions, because there will probably be some way for an 
interface to tell if the function at least conforms to the appropriate 
signature.  But for modules this will still be an issue.

I am not saying we shouldn't have a tp_conform; just suggesting that 
it may be appropriate for functions and modules (as well as classic 
classes) to have their tp_conform delegate back to 
self.__dict__['__conform__'] instead of a null implementation.
I have not considered conformance of such objects as functions or 
modules; if that is important, I need to add it to the reference 
implementation in the PEP.  I'm reluctant to just get __conform__ from 
the object, though; it leads to all sort of issues with a *class* 
conforming vs its *instances*, etc.  Maybe Guido can Pronounce a little 
on this sub-issue...


I don't see the benefit of LiskovViolation, or of doing the exact type 
check vs. the loose check.  What is the use case for these?  Is it to 
allow subclasses to say, "Hey I'm not my superclass?"  It's also a bit 
confusing to say that if the routines "raise any other exceptions" 
they're propagated.  Are you saying that LiskovViolation is *not* 
propagated?
Indeed I am -- I thought that was very clearly expressed!  
LiskovViolation means to skip the loose isinstance check, but it STILL 
allows explicitly registered adapter factories a chance (if somebody 
registers such an adapter factory, presumably they've coded a suitable 
adapter object type to deal with some deuced Liskov violation, see...). 
 On the other hand, if some random exception occurs in __conform__ or 
__adapt__, that's a bug somewhere, so the exception propagates in order 
to help debugging.  The previous version treated TypeError specially, 
but I think (on the basis of just playing around a bit, admittedly) 
that offers no real added value and sometimes will hide bugs.


If none of the first four mechanisms worked, as a last-ditch
attempt, 'adapt' falls back to checking a registry of adapter
factories, indexed by the protocol and the type of `obj', to meet
the fifth case.  Adapter factories may be dynamically registered
and removed from that registry to provide "third party adaptation"
of objects and protocols that have no knowledge of each other, in
a way that is not invasive to either the object or the protocols.
This should either be fleshed out to a concrete proposal, or dropped.  
There are many details that would need to be answered, such as whether 
"type" includes subtypes and whether it really means type or 
__class__.  (Note that isinstance() now uses __class__, allowing proxy 
objects to lie about their class; the adaptation system should support 
this too, and both the Zope and PyProtocols interface s

Re: [Python-Dev] PEP 246, redux

[Phillip J. Eby]

At 01:19 PM 1/10/05 -0500, Clark C. Evans wrote:
Alex,
  This is wonderful work, thank you for keeping the ball in the air;
  I'm honored to keep my name as a co-author -- kinda like a free lunch.
Phillip,
  Once again, thank you!  Without PyProtocols and your advocacy,
  this proposal might have been buried in the historical bit-bucket.
On Mon, Jan 10, 2005 at 12:43:44PM -0500, Phillip J. Eby wrote:
| -1 if this introduces a performance penalty to a wide range of
| adaptations (i.e. those using abstract base classes), just to support
| people who want to create deliberate Liskov violations.  I personally
| don't think that we should pander to Liskov violators, especially since
| Guido seems to be saying that there will be some kind of interface
| objects available in future Pythons.
I particularly like Alex's Liskov violation error; although it is
not hugely common, it does happen, and there should be a way for a
class to indicate that it's only being used for implementation.
Perhaps... if the class doesn't have a __conform__ method, then its
adaptation is automatic (that is, only the class can raise this
case).  The rationale for only enabling one of the two paths is that
the base class would have been in-place before the derived class was
created; therefore, it is highly unlikely that __adapt__ would ever
be of help.  Therefore, there might be a performance penalty, but it'd
be really small, simply checking to see if the slot is filled in.
The performance penalty I was talking about was for using an abstract base 
class, in a subclass with a __conform__ method for conformance to other 
protocols.  In this case, __conform__ will be uselessly called every time 
the object is adapted to the abstract base class.

IMO it's more desirable to support abstract base classes than to allow 
classes to "opt out" of inheritance when testing conformance to a base 
class.  If you don't have an "is-a" relationship to your base class, you 
should be using delegation, not inheritance.  (E.g. 'set' has-a 'dict', not 
'set' is-a 'dict', so 'adapt(set,dict)' should fail, at least on the basis 
of isinstance checking.)

The other problem with a Liskov opt-out is that you have to explicitly do a 
fair amount of work to create a LiskovViolation-raising subclass; that work 
would be better spent migrating to use delegation instead of inheritance, 
which would also be cleaner and more comprehensible code than writing a 
__conform__ hack to announce your bad style in having chosen to use 
inheritance where delegation is more appropriate.  ;)

This latter problem is actually much worse than the performance issue, 
which was just my initial impression.  Now that I've thought about it some 
more, I think I'm against supporting Liskov violations even if it were 
somehow *faster*.  :)

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Clark C. Evans]

Alex,

  This is wonderful work, thank you for keeping the ball in the air;
  I'm honored to keep my name as a co-author -- kinda like a free lunch.

Phillip,

  Once again, thank you!  Without PyProtocols and your advocacy,
  this proposal might have been buried in the historical bit-bucket.

On Mon, Jan 10, 2005 at 12:43:44PM -0500, Phillip J. Eby wrote:
| -1 if this introduces a performance penalty to a wide range of 
| adaptations (i.e. those using abstract base classes), just to support 
| people who want to create deliberate Liskov violations.  I personally 
| don't think that we should pander to Liskov violators, especially since 
| Guido seems to be saying that there will be some kind of interface 
| objects available in future Pythons.

I particularly like Alex's Liskov violation error; although it is
not hugely common, it does happen, and there should be a way for a 
class to indicate that it's only being used for implementation.

Perhaps... if the class doesn't have a __conform__ method, then its
adaptation is automatic (that is, only the class can raise this
case).  The rationale for only enabling one of the two paths is that
the base class would have been in-place before the derived class was
created; therefore, it is highly unlikely that __adapt__ would ever
be of help.  Therefore, there might be a performance penalty, but it'd 
be really small, simply checking to see if the slot is filled in.

Best,

Clark

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Phillip J. Eby]

At 12:43 PM 1/10/05 -0500, Phillip J. Eby wrote:
As a practical matter, all of the existing interface systems (Zope, 
PyProtocols, and even the defunct Twisted implementation) treat interface 
inheritance as guaranteeing substitutability for the base interface, and 
do so transitively.
An additional data point, by the way: the Eclipse Java IDE has an 
adaptation system that works very much like PEP 246 does, and it appears 
that in a future release they intend to support automatic adapter 
transitivity, so as to avoid requiring each provider of an interface to 
"provide O(n^2) adapters when writing the nth version of an 
interface."  IOW, their current release is transitive only for interface 
inheritance ala Zope or Twisted; their future release will be transitive 
for adapter chains ala PyProtocols.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 246, redux

[Phillip J. Eby]

At 03:42 PM 1/10/05 +0100, Alex Martelli wrote:
The fourth case above is subtle.  A break of substitutability can
occur when a subclass changes a method's signature, or restricts
the domains accepted for a method's argument ("co-variance" on
arguments types), or extends the co-domain to include return
values which the base class may never produce ("contra-variance"
on return types).  While compliance based on class inheritance
_should_ be automatic, this proposal allows an object to signal
that it is not compliant with a base class protocol.
-1 if this introduces a performance penalty to a wide range of adaptations 
(i.e. those using abstract base classes), just to support people who want 
to create deliberate Liskov violations.  I personally don't think that we 
should pander to Liskov violators, especially since Guido seems to be 
saying that there will be some kind of interface objects available in 
future Pythons.


Just like any other special method in today's Python, __conform__
is meant to be taken from the object's class, not from the object
itself (for all objects, except instances of "classic classes" as
long as we must still support the latter).  This enables a
possible 'tp_conform' slot to be added to Python's type objects in
the future, if desired.
One note here: Zope and PEAK sometimes use interfaces that a function or 
module may implement.  PyProtocols' implementation does this by adding a 
__conform__ object to the function's dictionary so that the function can 
conform to a particular signature.  If and when __conform__ becomes 
tp_conform, this may not be necessary any more, at least for functions, 
because there will probably be some way for an interface to tell if the 
function at least conforms to the appropriate signature.  But for modules 
this will still be an issue.

I am not saying we shouldn't have a tp_conform; just suggesting that it may 
be appropriate for functions and modules (as well as classic classes) to 
have their tp_conform delegate back to self.__dict__['__conform__'] instead 
of a null implementation.


The object may return itself as the result of __conform__ to
indicate compliance.  Alternatively, the object also has the
option of returning a wrapper object compliant with the protocol.
If the object knows it is not compliant although it belongs to a
type which is a subclass of the protocol, then __conform__ should
raise a LiskovViolation exception (a subclass of AdaptationError).
Finally, if the object cannot determine its compliance, it should
return None to enable the remaining mechanisms.  If __conform__
raises any other exception, "adapt" just propagates it.
To enable the third case, when the protocol knows about the
object, the protocol must have an __adapt__() method.  This
optional method takes two arguments:
- `self', the protocol requested
- `obj', the object being adapted
If the protocol finds the object to be compliant, it can return
obj directly.  Alternatively, the method may return a wrapper
compliant with the protocol.  If the protocol knows the object is
not compliant although it belongs to a type which is a subclass of
the protocol, then __adapt__ should raise a LiskovViolation
exception (a subclass of AdaptationError).  Finally, when
compliance cannot be determined, this method should return None to
enable the remaining mechanisms.  If __adapt__ raises any other
exception, "adapt" just propagates it.
The fourth case, when the object's class is a sub-class of the
protocol, is handled by the built-in adapt() function.  Under
normal circumstances, if "isinstance(object, protocol)" then
adapt() returns the object directly.  However, if the object is
not substitutable, either the __conform__() or __adapt__()
methods, as above mentioned, may raise an LiskovViolation (a
subclass of AdaptationError) to prevent this default behavior.
I don't see the benefit of LiskovViolation, or of doing the exact type 
check vs. the loose check.  What is the use case for these?  Is it to allow 
subclasses to say, "Hey I'm not my superclass?"  It's also a bit confusing 
to say that if the routines "raise any other exceptions" they're 
propagated.  Are you saying that LiskovViolation is *not* propagated?


If none of the first four mechanisms worked, as a last-ditch
attempt, 'adapt' falls back to checking a registry of adapter
factories, indexed by the protocol and the type of `obj', to meet
the fifth case.  Adapter factories may be dynamically registered
and removed from that registry to provide "third party adaptation"
of objects and protocols that have no knowledge of each other, in
a way that is not invasive to either the object or the protocols.
This should either be fleshed out to a concrete proposal, or 
dropped.  There are many details that would need to be answer

Re: [Python-Dev] PEP 246, redux

[Guido van Rossum]

> I had been promising to rewrite PEP 246 to incorporate the last several
> years' worth of discussions &c about it, and Guido's recent "stop the
> flames" artima blog post finally pushed me to complete the work.
> Feedback is of course welcome, so I thought I had better repost it
> here, rather than relying on would-be commenters to get it from CVS...

Thanks for doing this, Alex! I yet have to read the whole thing [will
attempt do so later today] but the few snippets I caught make me feel
this is a big step forward.

I'm wondering if someone could do a similar thing for PEP 245,
interfaces syntax? Alex hinted that it's a couple of rounds behind the
developments in Zope and Twisted. I'm personally not keen on needing
*two* new keywords (interface and implements) so I hope that whoever
does the rewrite could add a section on the advantages and
disadvantages of the 'implements' keyword (my simplistic alternative
proposal is to simply include interfaces in the list of bases in the
class statement; the metaclass can then sort it out).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

RE: [Python-Dev] Re: Subscribing to PEP updates

[Barry Warsaw]

On Mon, 2005-01-10 at 09:40, Michael Chermside wrote:
> Barry writes:
> > As an experiment, I just added a PEP topic to the python-checkins
> > mailing list.  You could subscribe to this list and just select the PEP
> > topic (which matches the regex "PEP" in the Subject header or first few
> > lines of the body).
> >
> > Give it a shot and let's see if that does the trick.
> 
> I just got notification of the change to PEP 246 (and I haven't received
> other checkin notifications), so I guess I can report that this is
> working.

Excellent!

> Thanks, Barry. Should we now mention this on c.l.py for others who
> may be interested?

Sure, I think that would be great.  Thanks.
-Barry



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

RE: [Python-Dev] os.removedirs() vs. shutil.rmtree()

[Batista, Facundo]

Title: RE: [Python-Dev] os.removedirs() vs. shutil.rmtree()





[Johannes Gijsbers]


#- So they're not identical, but I do agree they should be consolidated
#- and moved into one module. I'd say shutil, both because the os
#- module is already awfully crowded, and because these functions are
#- "high-level operations on files and collections of files" rather
#- than "a more portable way of using operating system dependent
#- functionality [...]".


+1.


We should be keeping this "should change this way" for when we restructure the std lib. There's already a wiki somewhere for this?

.    Facundo


Bitácora De Vuelo: http://www.taniquetil.com.ar/plog
PyAr - Python Argentina: http://pyar.decode.com.ar/



  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ADVERTENCIA.


La información contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener información confidencial o propietaria, cuya divulgación es sancionada por la ley.

Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no está autorizado a divulgar, copiar, distribuir o retener información (o parte de ella) contenida en este mensaje. Por favor notifíquenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magnético) que pueda haber realizado del mismo.

Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telefónica Comunicaciones Personales S.A. o alguna empresa asociada.

Los mensajes electrónicos pueden ser alterados, motivo por el cual Telefónica Comunicaciones Personales S.A. no aceptará ninguna obligación cualquiera sea el resultante de este mensaje.

Muchas Gracias.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 246, redux

[Alex Martelli]

I had been promising to rewrite PEP 246 to incorporate the last several 
years' worth of discussions &c about it, and Guido's recent "stop the 
flames" artima blog post finally pushed me to complete the work.  
Feedback is of course welcome, so I thought I had better repost it 
here, rather than relying on would-be commenters to get it from CVS... 
I'm also specifically CC'ing Clark, the co-author, since he wasn't 
involved in this rewrite and of course I owe it to him to change or 
clearly attribute to myself anything he doesn't like to have "under his 
own name"!

Thanks,
Alex
PEP: 246
Title: Object Adaptation
Version: $Revision: 1.6 $
Author: [EMAIL PROTECTED] (Alex Martelli),
[EMAIL PROTECTED] (Clark C. Evans)
Status: Draft
Type: Standards Track
Created: 21-Mar-2001
Python-Version: 2.5
Post-History: 29-Mar-2001, 10-Jan-2005
Abstract
This proposal puts forth an extensible cooperative mechanism for
the adaptation of an incoming object to a context which expects an
object supporting a specific protocol (say a specific type, class,
or interface).
This proposal provides a built-in "adapt" function that, for any
object X and any protocol Y, can be used to ask the Python
environment for a version of X compliant with Y.  Behind the
scenes, the mechanism asks object X: "Are you now, or do you know
how to wrap yourself to provide, a supporter of protocol Y?".
And, if this request fails, the function then asks protocol Y:
"Does object X support you, or do you know how to wrap it to
obtain such a supporter?"  This duality is important, because
protocols can be developed after objects are, or vice-versa, and
this PEP lets either case be supported non-invasively with regard
to the pre-existing component[s].
Lastly, if neither the object nor the protocol know about each
other, the mechanism may check a registry of adapter factories,
where callables able to adapt certain objects to certain protocols
can be registered dynamically.  This part of the proposal is
optional: the same effect could be obtained by ensuring that
certain kinds of protocols and/or objects can accept dynamic
registration of adapter factories, for example via suitable custom
metaclasses.  However, this optional part allows adaptation to be
made more flexible and powerful in a way that is not invasive to
either protocols or other objects, thereby gaining for adaptation
much the same kind of advantage that Python standard library's
"copy_reg" module offers for serialization and persistence.
This proposal does not specifically constrain what a protocol
_is_, what "compliance to a protocol" exactly _means_, nor what
precisely a wrapper is supposed to do.  These omissions are
intended to leave this proposal compatible with both existing
categories of protocols, such as the existing system of type and
classes, as well as the many concepts for "interfaces" as such
which have been proposed or implemented for Python, such as the
one in PEP 245 [1], the one in Zope3 [2], or the ones discussed in
the BDFL's Artima blog in late 2004 and early 2005 [3].  However,
some reflections on these subjects, intended to be suggestive and
not normative, are also included.
Motivation
Currently there is no standardized mechanism in Python for
checking if an object supports a particular protocol.  Typically,
existence of certain methods, particularly special methods such as
__getitem__, is used as an indicator of support for a particular
protocol.  This technique works well for a few specific protocols
blessed by the BDFL (Benevolent Dictator for Life).  The same can
be said for the alternative technique based on checking
'isinstance' (the built-in class "basestring" exists specifically
to let you use 'isinstance' to check if an object "is something
like a string").  Neither approach is easily and generally
extensible to other protocols, defined by applications and third
party frameworks, outside of the standard Python core.
Even more important than checking if an object already supports a
given protocol can be the task of obtaining a suitable adapter
(wrapper or proxy) for the object, if the support is not already
there.  For example, a string does not support the file protocol,
but you can wrap it into a StringIO instance to obtain an object
which does support that protocol and gets its data from the string
it wraps; that way, you can pass the string (suitably wrapped) to
subsystems which require as their arguments objects that are
readable as files.  Unfortunately, there is currently no general,
standardized way to automate this extremely important kind of
"adaptation by wrapping" operations.
Typically, today, when you pass objects to a context expecting a
particular protocol, either the object knows about the context an

RE: [Python-Dev] Re: Subscribing to PEP updates

[Michael Chermside]

Barry writes:
> As an experiment, I just added a PEP topic to the python-checkins
> mailing list.  You could subscribe to this list and just select the PEP
> topic (which matches the regex "PEP" in the Subject header or first few
> lines of the body).
>
> Give it a shot and let's see if that does the trick.

I just got notification of the change to PEP 246 (and I haven't received
other checkin notifications), so I guess I can report that this is
working.

Thanks, Barry. Should we now mention this on c.l.py for others who
may be interested?

-- Michael Chermside

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com