[Python-Dev] Re: PEP 642: Constraint Pattern Syntax for Structural Pattern Matching

[Gustavo Carneiro]

I think using symbols like ? and == in patterns looks stylistically ugly,
and unintuitive, albeit more explicit.

I, too, would rather have pattern matching more explicit, but it shouldn't
need to be so ugly (yes, I know, ugly is a subjective term, so be it).

I would propose that, opposite to what this PEP 642 proposes, matching as
literal terms should be the default, and a special notation should be used
for binding to names.

match number:
case 0:
print("Nothing")
case 1:
print("Just one")

This would be equivalent:

zero = 0
one = 1
match number:
case zero:
print("Nothing")
case one:
print("Just one")


And I would propose to use "as" for the notation of binding to a variable,
possibly in combination with "_" for the wildcard pattern:

expected_value = "xxx"
match some_json:
case {"foo": expected_value}:  # matches {"foo": "xxx"}
pass
case {"foo": _ as bar}:  # matches any {"foo": }
print(f"json got foo value {bar}")


Yes, I understand that being forced to use "_ as name" in a lot of patterns
is more verbose, but I posit that it is both explicit _and_ intuitive.  And
perhaps not as ugly as ? and ==.

In my mind, I don't see that this "as" usage causes any confusion with the
"as" in context managers.  That is a cop-out.  I see this "as" as more akin
to the exception handling:

try:
   ...
except RuntimeError as error:
   ...


See?  No context manager protocol involved here.  "as" is simply
representing a name binding.


On Sat, 31 Oct 2020 at 07:17, Nick Coghlan  wrote:

> Hi folks,
>
> This is a mailing list repost of the Discourse thread at
>
> https://discuss.python.org/t/pep-642-constraint-pattern-syntax-for-structural-pattern-matching/5614
>
> The rendered version of the PEP can be found here:
> https://www.python.org/dev/peps/pep-0642/
>
> The full text is also quoted in the Discourse thread.
>
> The remainder of this email is the same introduction that I posted on
> Discourse.
>
> I’m largely a fan of the Structural Pattern Matching proposal in PEP
> 634, but there’s one specific piece of the syntax proposal that I
> strongly dislike: the idea of basing the distinction between capture
> patterns and value patterns purely on whether they use a simple name
> or a dotted name.
>
> Thus PEP 642, which retains most of PEP 634 unchanged, but adjusts
> value checks to use an explicit prefix syntax (either `?EXPR` for
> equality constraints, or `?is EXPR` for identity constraints), rather
> than relying on users learning that literals and attribute lookups in
> a capture pattern mean a value lookup check, while simple names mean a
> capture pattern (unlike both normal expressions, where all three mean
> a value lookup, and assignment targets, where both simple and dotted
> names bind a new reference).
>
> The PEP itself has a lot of words explaining why I’ve made the design
> decisions I have, as well as the immediate and potential future
> benefits offered by using an explicit prefix syntax for value
> constraints, but the super short form goes like this:
>
> * if you don’t like match statements at all, or wish we were working
> on designing a C-style switch statement instead, then PEP 642 isn’t
> going to appeal to you any more than PEP 634 does
> * if, like me, you don’t like the idea of breaking the existing
> property of Python that caching the result of a value lookup
> subexpression in a local variable and then using that variable in
> place of the original subexpression should “just work”, then PEP 642’s
> explicit constraint prefix syntax may be more to your liking
> * however, if the idea of the `?` symbol becoming part of Python’s
> syntax doesn’t appeal to you, then you may consider any improved
> clarity of intent that PEP 642 might offer to not be worth that cost
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WT3ZZ42XJ4G6Y3H26RWUFN5GCUIFLMQ7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3Q5KSGEXWGNC5JNUIXNTZJB4Q36IM3DY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Pattern matching reborn: PEP 622 is dead, long live PEP 634, 635, 636

[Gustavo Carneiro]

It's true that asyncio.wait provides the tools that you need, but it's a
bit clunky to use correctly.

Maybe would be something along the lines of:

--
queue1 = asyncio.Queue()
queue2 = asyncio.Queue()
...
get1 = asyncio.create_task(queue1.get())
get2 = asyncio.create_task(queue2.get())
await asyncio.wait({get1, get2}, return_when=asyncio.FIRST_COMPLETED)
match [task.done() for task in (get1, get2)]:
case [True, False]:  get2.cancel(); item1 = await get1; 
case [False, True]:  get1.cancel(); item2 = await get2; 
case [True, True]:  item1 = await get1; ; item2 = await get2; 
--

If asyncio.Queue() is the equivalent of Go channels, perhaps it would be
worth designing a new API for asyncio.Queue, one that is better suited to
the match statement:

class Queue:
   async def read_wait(self) -> 'Queue':
   """
   Waits until the queue has at least one item ready to read, without
actually consuming the item.
   """

Then we could more easily use match statement with multiple queues, thus:

--
async def ready_queue(*queues: asyncio.Queue) -> asyncio.Queue:
   """
   Take multiple queue parameters and waits for at least one of them to
have items pending to read, returning that queue.
   """
   await asyncio.wait({queue.read_wait() for queue in queues},
return_when=asyncio.FIRST_COMPLETED)
   for queue in queues:
  if queue.qsize() > 0:
  return queue

...

queue1 = asyncio.Queue()
queue2 = asyncio.Queue()

...

match await ready_queue(queue1, queue2):
case queue1:  item1 = queue1.get_nowait(); 
case queue2:  item2 = queue2.get_nowait(); 
--

Which is less clunky, maybe?...

The above is not 100% bug free.  I think those queue.get_nowait() calls may
still end up raising QueueEmpty exceptions, in case there is another
concurrent reader for those queues.  This code would need more work, most
likely.

In any case, perhaps it's not the match statement that needs to change, but
rather asyncio API that needs to be enhanced.


On Sun, 25 Oct 2020 at 01:14, Nick Coghlan  wrote:

> On Sat., 24 Oct. 2020, 4:21 am Guido van Rossum,  wrote:
>
>> On Fri, Oct 23, 2020 at 6:19 AM Tin Tvrtković 
>> wrote:
>>
>>> Hi,
>>>
>>> first of all, I'm a big fan of the changes being proposed here since in
>>> my code I prefer the 'union' style of logic over the OO style.
>>>
>>> I was curious, though, if there are any plans for the match operator to
>>> support async stuff. I'm interested in the problem of waiting on multiple
>>> asyncio tasks concurrently, and having a branch of code execute depending
>>> on the task.
>>>
>>> Currently this can be done by using asyncio.wait, looping over the done
>>> set and executing an if-else chain there, but this is quite tiresome. Go
>>> has a select statement (https://tour.golang.org/concurrency/5) that
>>> looks like this:
>>>
>>> select {
>>> case <-ch1:
>>> fmt.Println("Received from ch1")
>>> case <-ch2:
>>> fmt.Println("Received from ch2")
>>> }
>>>
>>> Speaking personally, this is a Go feature I miss a lot when writing
>>> asyncio code. The syntax is similar to what's being proposed here. Although
>>> it could be a separate thing added later, async match, I guess.
>>>
>>
>> Hadn't seen this before. You could propose this as a follow-up for 3.11.
>> But aren't Go channels more like asyncio Queues? I guess we'd need way more
>> in terms of a worked-out example (using asyncio code, not Go code).
>>
>
> I think we'd also want to see how far folks get with using guard clauses
> for this kind of "where did the data come from?" check - the only
> specifically asynchronous bit would be the "await multiple tasks"
> operation, and you can already tell asyncio.wait() to return on the first
> completed task rather than waiting for all the results.
>
> Cheers,
> Nick.
>
>
>
>> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/NQWYLFLGLLCEHAXYHUOXQ3M7IOEL65ET/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BHHNI54W6PVET3RD7XVHNOHFUAEDSVS5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622 version 2 (Structural Pattern Matching)

[Gustavo Carneiro]

On Fri, 17 Jul 2020 at 12:26,  wrote:

> Hello everyone,
>
> I'm sorry if my proposition has already being said, or even withdrawn,
> but I think that capture variables shouldn't be as implicit as they
> are now. I didn't see any mention of capture variable patterns in
> the rejected ideas. So here is my idea:
>
> I've looked at the PEP very quickly, jumping on the examples to have
> a taste and an idea of what was going here. I saw a new kind of control
> structure based on structural pattern matching (pattern based on
> classes or compositions of classes to make it short). A very good
> idea, emphasized by Tobias Kohn ("Another take on PEP 622") is that
> pattern matching eases the writting of code based on matching such
> structures, and that capturing values stored inside of these
> structures at the match time really eases the writting of the code
> associated with this match.
>
> But... looking at the examples, it wasn't very obvious that some
> variables were catching variables and some others were matching ones.
> I then read in details some rules about how to discover what is a
> captured variable. But I'm not sure everybody will do this...
>
> Zen of Python tells us that "explicit is better than implicit". I know
> this is not a rule written in the stone, but I think here, it applies
> very well.
>
> Guido said :
> > We’re really looking
> > for a solution that tells you when you’re looking at an individual
> > case which variables are captured and which are used for
> > load-and-compare.
> >
> > Marking up the capture variables with some sigil (e.g. $x or
> > x?)
> > or other markup (e.g. backticks or ) makes this common case ugly
> > and inconsistent: it’s unpleasant to see for example
> >
> > case %x, %y:
> > print(x, y)
>
> Guido talk about a "sigil", which seems to be a meaningless mark only
> here to help the parser understand what the dev was writing.
>
> I propose that this "sigil" be the affectation mark : "=". Look :
>
> z = 42
> match pt:
> case x=, y=, z:
> print(x, y, "z == 42")
>
> Or this one :
>
> def make_point_3d(pt):
> match pt:
> case (x=, y=):
> return Point3d(x, y, 0)
> case (x=, y=, z=):
> return Point3d(x, y, z)
> case Point2d(x=, y=):
> return Point3d(x, y, 0)
> case Point3d(_, _, _):
> return pt
> case _:
> raise TypeError("not a point we support")
>
>
I kind of agree it is nicer to be more explicit.  But somehow x= looks
ugly. It occurred to me (and, again, apologies if already been mentioned),
we might use the `as` keyword here.

The example above would become:

def make_point_3d(pt):
match pt:
case (as x, as y):
return Point3d(x, y, 0)
case (as x, as y, as z):
return Point3d(x, y, z)
case Point2d(as x, as y):
return Point3d(x, y, 0)
case Point3d(_, _, _):
return pt
case _:
raise TypeError("not a point we support")

If having "as x" as a standalone expression without anything to the left of
"as" causes confusion, we could instead mandate the use of _ thus:

case (_ as x, _ as y):
return Point3d(x, y, 0)

On the need to be explicit:
>
> Simple case blocks will perhaps be a bit longer to write, but will
> not be harder to read, since they stay in the "simple case blocks"
> family.
>
> More complex cases will be harder to write, but the explicit markup
> will help understand what will be captured and where, and what will
> be looked-and-matched, using already known rules : looked-and-matched
> expressions will be computed as usual, then compared with the match
> term, and captured expression will be computed to a l-value (which
> is much more restrictive than random expressions).
>
> Moreover, explicilty introducing a difference between "capture" and
> "look-and-match" will help newcomers to understand what is the point
> about a match without they have to look at a PEP or other normative
> document.
>
> Remember that code has to be readable, because it will be read much
> more often than written. The reader has to understand quickly but not
> in details what will happen. Being explicit removes they the task
> to concentrate on this point.
>
> Also remember that Python has to be teached, and that all that is
> implicit in the code have to be explicited when teaching. And the
> longer you teach microdetails like what is the difference between
> "capture" vs "look-and-match", the less your audience will be prone
> to listen to you.
>
> On the drawback to be explicit:
>
> Adding a mark to every captured variables can and will be annoying.
> More annoying than not adding it. It's obvious. But we don't expect to
> have more than a handful of captured variables per case. Or the case
> is perhaps too complex, not 

[Python-Dev] Re: PEP 622 version 2 (Structural Pattern Matching)

[Gustavo Carneiro]

On Fri, 10 Jul 2020 at 10:33, Glenn Linderman  wrote:

> On 7/10/2020 1:21 AM, Stefano Borini wrote:
> > Just my 2 cents, I find it kind of annoying that the whole structure
> > requires two levels of indentation to actually reach the operational
> > code.
> > This would be a first in python.
> >
> > I would prefer an option akin to if elif elif else where each block is
> > only one level deep.
> Me too.
>
> That would also sidestep the dilemma of whether else: (if implemented)
> should be indented like case: or like match: because they would be the
> same.
>
> match:
>  t
> case ("rect", real, imag):
>  return complex(real, imag)
> case ("polar", r, phi):
>  return complex( r* cos(phi), r*sin(phi)
> else:
>  return None
>
> but it does make the match: block not a statement group, which was
> disturbing to some.
>
> On the other hand, this has a correspondence to:
>
> try:
>   throw expression
> except (type of expression) as exc1:
>   blah blah1
> except (another type) as exc2:
>  blah blah2
> else:
>  blah blah3
>

The problem of the try...except structure, with less indentation, is that,
yes, it is OK for exceptions because normally you have 2 or 3 `except XXX`
clauses, therefore it is usually easy to follow, if the number of vertical
lines in the entire block of try-catch is low enough.

But I have had cases with catching many exception types, each with its own
block of 4 or 5 lines, adding up to a block of try-excepts that doesn't
even fit in a single window of my editor.  In that case, I always have
wished for except clauses to be extra indented, to more easily distinguish
where the try..except block ends.

Therefore, I posit that the style of try...except indentation only works
where the number of cases is small.

But for the case of pattern matching, I expect the number of cases to be
matched to be a lot higher than exception handling cases.  Having cases to
be matched be indented is, IMHO, a nice visual cue to help the reader
understand where the pattern matching block ends.


> In fact, one _could_ wrap this whole feature into the try: syntax... the
> match statement would be tried, and the cases would be special types of
> exception handlers:
>
> try:
>  match expression
> case ("rect", real, imag):
>  return complex(real, imag)
> case ("polar", r, phi):
>  return complex( r* cos(phi), r*sin(phi)
> else:
>  return None
>
> If the expression could fail to be calculated, one could have a mix of
> except clauses also to catch those, rather than needing to wrap the
> whole match expression in a separate try to handle that case [making the
> nesting even deeper :( ]
>
> There might even be a use for using case clauses to extend "normal"
> exception handling, where the exception object could be tested for its
> content as well as its class to have different handling.
>
> try:
>  raise Exception("msg", 35, things)
> case Exception( x, "widgets"):
>  blah blah 1
> case Exception( x, "characters"):
>  blah blah 2
> else:
>  blah blah 3
>
> In this not-fully-thought-through scenario, maybe the keyword match
> isn't even needed: "raise expression" could do the job, or they could be
> aliases to signify intent.
>
> In other words, a match expression would always "fail". The only
> mismatch here is that it points out the difference between try-else and
> match-else: try-else is executed if there is no failure, but if match
> always fails, else would never be appropriate, and case _: would be.
>
> In any case, it does seem there is a strong correlation between match
> processing and try processing, that I didn't see during other
> discussions of the possible structural similarities. "match 3 / 0:"
> would clearly need to be wrapped in a try:
>
> try:
>  match x / y:
>   case 43:
> print("wow, it is 43")
>   case 22:
> print("22 seemed less likely than 43 for some reason")
>  case _:
>print("You get what you get")
> except ZeroDivisionError as exc:
>  print(f"But sometimes you get an exception {exc}")
>
> or:
>
> try:
>  raise x / y
> case 43:
>  print("wow, it is 43")
> case 22:
>  print("22 seemed less likely than 43 for some reason")
> case exc := ZeroDivisionError:
>  print(f"But sometimes you get an exception: {exc}")
> case _:
>  print("You get what you get")
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GDP2KKB3SUWQZRSNTR5N36LXZ6HDS2QL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing 

[Python-Dev] Re: PEP 622 version 2 (Structural Pattern Matching)

[Gustavo Carneiro]

*facepalm* this is right there in the PEP, already, as one possible
alternative.  Apologies for the noise. :-/

On Wed, 8 Jul 2020 at 19:27, Gustavo Carneiro  wrote:

>
>
> On Wed, 8 Jul 2020 at 16:05, Guido van Rossum  wrote:
>
>> Today I’m happy (and a little trepidatious) to announce the next
>> version of PEP 622, Pattern Matching. As authors we welcome Daniel F
>> Moisset in our midst. Daniel wrote a lot of the new text in this
>> version, which introduces the subject matter much more gently than the
>> first version did. He also convinced us to drop the `__match__`
>> protocol for now: the proposal stands quite well without that kind of
>> extensibility, and postponing it will allow us to design it at a later
>> time when we have more experience with how `match` is being used.
>>
>> That said, the new version does not differ dramatically in what we
>> propose. Apart from dropping `__match__` we’re dropping the leading
>> dot to mark named constants, without a replacement, and everything
>> else looks like we’re digging in our heels. Why is that? Given the
>> firestorm of feedback we received and the numerous proposals (still
>> coming) for alternative syntax, it seems a bad tactic not to give up
>> something more substantial in order to get this proposal passed. Let
>> me explain.
>>
>> Language design is not like politics. It’s not like mathematics
>> either, but I don’t think this situation is at all similar to
>> negotiating a higher minimum wage in exchange for a lower pension,
>> where you can definitely argue about exactly how much lower/higher
>> you’re willing to go. So I don’t think it’s right to propose making
>> the feature a little bit uglier just to get it accepted.
>>
>> Frankly, 90% of the issue is about what among the authors we’ve dubbed
>> the “load/store” problem (although Tobias never tires to explain that
>> the “load” part is really “load-and-compare”). There’s a considerable
>> section devoted to this topic in the PEP, but I’d like to give it
>> another try here.
>>
>> In case you’ve been avoiding python-dev lately, the problem is
>> this. Pattern matching lets you capture values from the subject,
>> similar to sequence unpacking, so that you can write for example
>> ```
>> x = range(4)
>> match x:
>> case (a, b, *rest):
>> print(f"first={a}, second={b}, rest={rest}")  # 0, 1, [2, 3]
>> ```
>> Here the `case` line captures the contents of the subject `x` in three
>> variables named `a`, `b` and `rest`. This is easy to understand by
>> pretending that a pattern (i.e., what follows `case`) is like the LHS
>> of an assignment.
>>
>> However, in order to make pattern matching more useful and versatile,
>> the pattern matching syntax also allows using literals instead of
>> capture variables. This is really handy when you want to distinguish
>> different cases based on some value, for example
>> ```
>> match t:
>> case ("rect", real, imag):
>> return complex(real, imag)
>> case ("polar", r, phi):
>> return complex(r * cos(phi), r * sin(phi))
>> ```
>> You might not even notice anything funny here if I didn’t point out
>> that `"rect"` and `"polar"` are literals -- it’s really quite
>> natural for patterns to support this once you think about it.
>>
>> The problem that everybody’s been concerned about is that Python
>> programmers, like C programmers before them, aren’t too keen to have
>> literals like this all over their code, and would rather give names to
>> the literals, for example
>> ```
>> USE_POLAR = "polar"
>> USE_RECT = "rect"
>> ```
>> Now we would like to be able to replace those literals with the
>> corresponding names throughout our code and have everything work like
>> before:
>> ```
>> match t:
>> case (USE_RECT, real, imag):
>> return complex(real, imag)
>> case (USE_POLAR, r, phi):
>> return complex(r * cos(phi), r * sin(phi))
>> ```
>>
>
> Forgive the intrusion, in case this wasn't already mentioned (I only read
> a fraction of emails on this), we could say that name enclosed in
> parenthesis would mean loading a constant, instead of storing in a variable:
>
> match t:
> case ((USE_RECT), real, imag):  # matches the constant USE_RECT
> literal value
> return complex(real, imag)
> case (USE_POLAR, r, phi):  # the USE_POLAR portion matc

[Python-Dev] Re: PEP 622 version 2 (Structural Pattern Matching)

[Gustavo Carneiro]

On Wed, 8 Jul 2020 at 16:05, Guido van Rossum  wrote:

> Today I’m happy (and a little trepidatious) to announce the next
> version of PEP 622, Pattern Matching. As authors we welcome Daniel F
> Moisset in our midst. Daniel wrote a lot of the new text in this
> version, which introduces the subject matter much more gently than the
> first version did. He also convinced us to drop the `__match__`
> protocol for now: the proposal stands quite well without that kind of
> extensibility, and postponing it will allow us to design it at a later
> time when we have more experience with how `match` is being used.
>
> That said, the new version does not differ dramatically in what we
> propose. Apart from dropping `__match__` we’re dropping the leading
> dot to mark named constants, without a replacement, and everything
> else looks like we’re digging in our heels. Why is that? Given the
> firestorm of feedback we received and the numerous proposals (still
> coming) for alternative syntax, it seems a bad tactic not to give up
> something more substantial in order to get this proposal passed. Let
> me explain.
>
> Language design is not like politics. It’s not like mathematics
> either, but I don’t think this situation is at all similar to
> negotiating a higher minimum wage in exchange for a lower pension,
> where you can definitely argue about exactly how much lower/higher
> you’re willing to go. So I don’t think it’s right to propose making
> the feature a little bit uglier just to get it accepted.
>
> Frankly, 90% of the issue is about what among the authors we’ve dubbed
> the “load/store” problem (although Tobias never tires to explain that
> the “load” part is really “load-and-compare”). There’s a considerable
> section devoted to this topic in the PEP, but I’d like to give it
> another try here.
>
> In case you’ve been avoiding python-dev lately, the problem is
> this. Pattern matching lets you capture values from the subject,
> similar to sequence unpacking, so that you can write for example
> ```
> x = range(4)
> match x:
> case (a, b, *rest):
> print(f"first={a}, second={b}, rest={rest}")  # 0, 1, [2, 3]
> ```
> Here the `case` line captures the contents of the subject `x` in three
> variables named `a`, `b` and `rest`. This is easy to understand by
> pretending that a pattern (i.e., what follows `case`) is like the LHS
> of an assignment.
>
> However, in order to make pattern matching more useful and versatile,
> the pattern matching syntax also allows using literals instead of
> capture variables. This is really handy when you want to distinguish
> different cases based on some value, for example
> ```
> match t:
> case ("rect", real, imag):
> return complex(real, imag)
> case ("polar", r, phi):
> return complex(r * cos(phi), r * sin(phi))
> ```
> You might not even notice anything funny here if I didn’t point out
> that `"rect"` and `"polar"` are literals -- it’s really quite
> natural for patterns to support this once you think about it.
>
> The problem that everybody’s been concerned about is that Python
> programmers, like C programmers before them, aren’t too keen to have
> literals like this all over their code, and would rather give names to
> the literals, for example
> ```
> USE_POLAR = "polar"
> USE_RECT = "rect"
> ```
> Now we would like to be able to replace those literals with the
> corresponding names throughout our code and have everything work like
> before:
> ```
> match t:
> case (USE_RECT, real, imag):
> return complex(real, imag)
> case (USE_POLAR, r, phi):
> return complex(r * cos(phi), r * sin(phi))
> ```
>

Forgive the intrusion, in case this wasn't already mentioned (I only read a
fraction of emails on this), we could say that name enclosed in parenthesis
would mean loading a constant, instead of storing in a variable:

match t:
case ((USE_RECT), real, imag):  # matches the constant USE_RECT
literal value
return complex(real, imag)
case (USE_POLAR, r, phi):  # the USE_POLAR portion matches anything
and stores in a USE_POLAR variable
return complex(r * cos(phi), r * sin(phi))

Advantages: in Python (and most programming languages), (x) is the same
thing as x.  So, no new syntax, or weird symbols, need to be introduced.

But the parser can distinguish (I hope), and guide the match statement
generation to the appropriate behaviour.

Yes, it's more typing, but hopefully the case is uncommon enough that the
extra typing is not a lot of burden.

Yes, it's easy to type USE_RECT when you really meant (USE_RECT).
Hopefully linters can catch this case and warn you.

OK, that's it, I just thought it was worth throwing yet another idea into
the pot :-)

-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing 

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

[Gustavo Carneiro]

On Wed, 24 Jun 2020 at 17:22, Victor Stinner  wrote:

[...]


> The question becomes: how to promote the limited C API? Should it
> become the default, rather than an opt-in option?
>

It would be interesting to find out what is the performance impact of using
limited C API, vs normal API, on some popular extensions.  This is
something that I wish had been included in PEP 384.

It would be great if the limited API could be the default, as it allows
building extensions once that work across most python versions.

-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BS7QTLNLEMCODXPEGAIRENGYR24JBZK3/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Gustavo Carneiro]

On Thu, 14 Feb 2019 at 15:52, Victor Stinner  wrote:

> Le jeu. 14 févr. 2019 à 14:38, Matthias Klose  a écrit :
> > Debian's concern about pointing python to python3 is that it will break
> software
> > after an upgrade.  The current state seems is still the same that Debian
> doesn't
> > want to ship a python symlink after the Python2 removal.
>
> The other safer alternative is to start to provide "py" launcher on
> Unix as well. Since it's something new, it's perfectly fine to decide
> from the start to make it point to the latest Python version by
> default.
>

While I like very much the idea of having `py` as command, does it really
need to be a wrapper command?  Why can't it simply be a symlink?

/usr/bin/py -> /usr/bin/python3

I worry about (1) startup time overhead of starting another process, (2)
added complexity of learning about py's additional command-line options, we
don't really need them, imho.


> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
>


-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: intended scope of assignment expression

[Gustavo Carneiro]

On Thu, 5 Jul 2018 at 13:43, Victor Stinner  wrote:

> Hi,
>
> My work (*) in the "Assignment expression and coding style: the while
> True case" thread helped me to understand something about the
> *intended* scope.
>
> While technically, assignment expressions keep the same scoping rules
> than assignment statements, writing "if (x := func()): ..." or "while
> (x := func()): ..." shows the "intented" scope of the variable. Even
> if, as explained properly in the PEP, the scope is wider (for good
> reasons) as "for line in file: ..." keeps line alive after the loop
> (nothing new under the sun). It's something subtle that I missed at
> the first read (of the code and the PEP), the difference is not
> obvious.
>
> x = func()
> if x:
> ... # obviously use x
> # do we still plan to use x here?
> # it's non obvious just by reading the if
>
> versus
>
> if (x := func()):
> ... # obviously use x
> # ":=" in the if "announces" that usually x is no longer used
> # here, even if technically x is still defined
>

I don't know if you're trying to propose something clever here, like "if (x
:= func()):" would assign to 'x' only inside the "then" body of the if, but
IMHO that would be a terrible idea:

1. it makes AE harder to explain.  Right now you can simply say AE works
just like any regular assignment, except that you can use in an
expression.  If you start adding exceptions to the rule, things get harder
and harder to explain;
2. it would probably have a cost in terms of performance: you'd have to
create another scope for every "then" body of any if statement that
contains an AE.  And, again, more scopes = things harder to understand.

-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A more flexible task creation

[Gustavo Carneiro]

On Fri, 15 Jun 2018 at 09:18, Michel Desmoulin 
wrote:

>
> >
> > The strict API compatibility requirements of core Python stdlib, coupled
> > with the very long feature release life-cycles of Python, make me think
> > this sort of thing perhaps is better built in an utility library on top
> > of asyncio, rather than inside asyncio itself?  18 months is a long long
> > time to iterate on these features.  I can't wait for Python 3.8...
> >
>
> A lot of my late requests come from my attempt to group some of that in
> a lib: https://github.com/Tygs/ayo
>

Ah, good idea.


> Most of it works, although I got read of context() recently, but the
> lazy task part really fails.
>
>
> Indeed, the API allows to do:
>
> async with ayo.scope() as run:
> task_list = run.all(foo(), foo(), foo())
> run.asap(bar())
> await task_list.gather()
> run.asap(baz())
>
>
>
> scope() return a nursery like object, and this works perfectly, with the
> usual guaranty of Trio's nursery, but working in asyncio right now.
>

To be honest, I see "async with" being abused everywhere in asyncio,
lately.  I like to have objects with start() and stop() methods, but
everywhere I see async context managers.

Fine, add nursery or whatever, but please also have a simple start() /
stop() public API.

"async with" is only good for functional programming.  If you want to go
more of an object-oriented style, you tend to have start() and stop()
methods in your classes, which will call start() & stop() (or close())
methods recursively on nested resources.  So of the libraries (aiopg, I'm
looking at you) don't support start/stop or open/close well.


> However, I tried to add to the mix:
>
> async with ayo.scope(max_concurrency=2) as run:
> task_list = run.all(foo(), foo(), foo())
> run.asap(bar())
> await task_list.gather()
> run.asap(baz())
>
> And I can get it to work. task_list will right now contains a list of
> tasks and None, because some tasks are not scheduled immediately. That's
> why I wanted lazytasks. I tried to create my own lazy tasks, but it
> never really worked. I'm going to try to go down the road of wrapping
> the unscheduled coro in a future-like object as suggested by Yuri. But
> having that built-in seems logical, elegant, and just good design in
> general: __init__ should not have side effects.
>

I tend to slightly agree, but OTOH if asyncio had been designed to not
schedule tasks automatically on __init__ I bet there would have been other
users complaining that "why didn't task XX run?", or "why do tasks need a
start() method, that is clunky!".  You can't please everyone...

Also, in
 task_list = run.all(foo(), foo(), foo())

As soon as you call foo(), you are instantiating a coroutine, which
consumes memory, while the task may not even be scheduled for a long time
(if you have 5000 potential tasks but only execute 10 at a time, for
example).

But if you do as Yuri suggested, you'll instead accept a function
reference, foo, which is a singleton, you can have many foo references to
the function, but they will only create coroutine objects when the task is
actually about to be scheduled, so it's more efficient in terms of memory.

-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A more flexible task creation

[Gustavo Carneiro]

On Thu, 14 Jun 2018 at 17:40, Tin Tvrtković  wrote:

> Hi,
>
> I've been using asyncio a lot lately and have encountered this problem
> several times. Imagine you want to do a lot of queries against a database,
> spawning 1 tasks in parallel will probably cause a lot of them to fail.
> What you need in a task pool of sorts, to limit concurrency and do only 20
> requests in parallel.
>
> If we were doing this synchronously, we wouldn't spawn 1 threads using
> 1 connections, we would use a thread pool with a limited number of
> threads and submit the jobs into its queue.
>
> To me, tasks are (somewhat) logically analogous to threads. The solution
> that first comes to mind is to create an AsyncioTaskExecutor with a
> submit(coro, *args, **kwargs) method. Put a reference to the coroutine and
> its arguments into an asyncio queue. Spawn n tasks pulling from this queue
> and awaiting the coroutines.
>

> It'd probably be useful to have this in the stdlib at some point.
>

Probably a good idea, yes, because it seems a rather common use case.

OTOH, I did something similar but for a different use case.  In my case, I
have a Watchdog class, that takes a list of (coro, *args, **kwargs).  What
it does is ensure there is always a task for each of the co-routines
running, and watches the tasks, if they crash they are automatically
restarted (with logging).  Then there is a stop() method to cancel the
watchdog-managed tasks and await them.  My use case is because I tend to
write a lot of singleton-style objects, which need book keeping tasks, or
redis pubsub listening tasks, and my primary concern is not starting lots
of tasks, it is that the few tasks I have must be restarted if they crash,
forever.

This is why I think it's not that hard to write "sugar" APIs on top of
asyncio, and everyone's needs will be different.

The strict API compatibility requirements of core Python stdlib, coupled
with the very long feature release life-cycles of Python, make me think
this sort of thing perhaps is better built in an utility library on top of
asyncio, rather than inside asyncio itself?  18 months is a long long time
to iterate on these features.  I can't wait for Python 3.8...


>
> Date: Wed, 13 Jun 2018 22:45:22 +0200
>> From: Michel Desmoulin 
>> To: python-dev@python.org
>> Subject: [Python-Dev] A more flexible task creation
>> Message-ID: 
>> Content-Type: text/plain; charset=utf-8
>>
>> I was working on a concurrency limiting code for asyncio, so the user
>> may submit as many tasks as one wants, but only a max number of tasks
>> will be submitted to the event loop at the same time.
>>
>> However, I wanted that passing an awaitable would always return a task,
>> no matter if the task was currently scheduled or not. The goal is that
>> you could add done callbacks to it, decide to force schedule it, etc
>>
>> I dug in the asyncio.Task code, and encountered:
>>
>> def __init__(self, coro, *, loop=None):
>> ...
>> self._loop.call_soon(self._step)
>> self.__class__._all_tasks.add(self)
>>
>> I was surprised to see that instantiating a Task class has any side
>> effect at all, let alone 2, and one of them being to be immediately
>> scheduled for execution.
>>
>> I couldn't find a clean way to do what I wanted: either you
>> loop.create_task() and you get a task but it runs, or you don't run
>> anything, but you don't get a nice task object to hold on to.
>>
>> I tried several alternatives, like returning a future, and binding the
>> future awaiting to the submission of a task, but that was complicated
>> code that duplicated a lot of things.
>>
>> I tried creating a custom task, but it was even harder, setting a custom
>> event policy, to provide a custom event loop with my own create_task()
>> accepting parameters. That's a lot to do just to provide a parameter to
>> Task, especially if you already use a custom event loop (e.g: uvloop). I
>> was expecting to have to create a task factory only, but task factories
>> can't get any additional parameters from create_task()).
>>
>> Additionally I can't use ensure_future(), as it doesn't allow to pass
>> any parameter to the underlying Task, so if I want to accept any
>> awaitable in my signature, I need to provide my own custom
>> ensure_future().
>>
>> All those implementations access a lot of _private_api, and do other
>> shady things that linters hate; plus they are fragile at best. What's
>> more, Task being rewritten in C prevents things like setting self._coro,
>> so we can only inherit from the pure Python slow version.
>>
>> In the end, I can't even await the lazy task, because it blocks the
>> entire program.
>>
>> Hence I have 2 distinct, but independent albeit related, proposals:
>>
>> - Allow Task to be created but not scheduled for execution, and add a
>> parameter to ensure_future() and create_task() to control this. Awaiting
>> such a task would just do like asyncio.sleep(O) until it is scheduled
>> for execution.
>>
>> - 

Re: [Python-Dev] PEP 572: Why not := as standard assignment operator?

[Gustavo Carneiro]

On 26 April 2018 at 16:18, Chris Angelico  wrote:

> On Thu, Apr 26, 2018 at 11:13 PM, Martin Teichmann
>  wrote:
> > Hi list,
> >
> > when reading PEP 572 I actually liked it a lot - I think it's actually
> > a cool idea. I think it's actually that cool an idea that it should be
> > made the default way of doing an assignment, over time phasing out the
> > good ole =.
> >
> > This would have several benefits:
> >
> > - people wouldn't have to worry about two different options
> > - different things would have a different look: assignment is :=,
> > keyword args is =, while comparison is ==. Especially beginners would
> > benefit from this clarity.
> >
> > in this case, for sure, we should make it possible to chain :=s, for
> > example by making it bind right-to-left, so that  a := b := 3 would be
> > a := (b := 3)
> >
> > I'm sorry if somebody brought that up already, but the discussion has
> > grown so huge that I couldn't read through it entirely.
>
> It has indeed grown huge. And in the interests of not growing it even
> huger, I'm not going to rehash the arguments against making := into
> the one and only operator, save to say one thing: there's no way that
> "x = 1" can be removed from the language any time soon, and by "soon"
> I mean even by the Yes Prime Minister definition, where "any day now",
> in strategic terms, meant "within the next half century".
>

In the interest of that, do you think := can be made illegal, by the
grammar, if used outside an expression?

a = 1  # legal
a := 1  # Syntax error
if a := 1:  # legal

Thanks in advance.

-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Gustavo Carneiro]

Big +1 to turning warnings on by default again.

When this behaviour first started, first I was surprised, then annoyed that
warnings were being suppressed.  For a few years I learned to have `export
PYTHONWARNINGS=default` in my .bashrc.

But eventually, the warnings in 3rd-party Python modules gradually
increased because, since warnings are disabled by default, authors of
command-line tools, or even python modules,  don't even realise they are
triggering the warning.

So developers stop fixing warnings because they are hidden.  Things get
worse and worse over the years.  Eventually I got fed up and removed the
PYTHONWARNINGS env var.

Showing warnings by default is good:
 1. End users who don't understand what those warnings are are unlikely to
see them since they don't command-line tools at all;
 2. The users that do see them are sufficiently proficient to be able to
submit a bug report;
 3. If you file a bug report in tool that uses a 3rd party module, the
author of that tool should open a corresponding bug report on the 3rd party
module that actually caused the warning;
 4. Given time, all the bug reports trickle down and create pressure on
module maintainers to fix warnings;
 5. If a module is being used and has no maintainer, it's a good indication
it is time to fork it or find an alternative.

Not fixing warnings is a form of technical debt that we should not
encourage.  It is not the Python way.


On 6 November 2017 at 02:05, Nick Coghlan  wrote:

> On the 12-weeks-to-3.7-feature-freeze thread, Jose Bueno & I both
> mistakenly though the async/await deprecation warnings were missing
> from 3.6.
>
> They weren't missing, we'd just both forgotten those warnings were off
> by default (7 years after the change to the default settings in 2.7 &
> 3.2).
>
> So my proposal is simple (and not really new): let's revert back to
> the way things were in 2.6 and earlier, with DeprecationWarning being
> visible by default, and app devs having to silence it explicitly
> during application startup (before they start importing third party
> modules) if they don't want their users seeing it when running on the
> latest Python version (e.g. this would be suitable for open source
> apps that get integrated into Linux distros and use the system Python
> there).
>
> This will also restore the previously clear semantic and behavioural
> different between PendingDeprecationWarning (hidden by default) and
> DeprecationWarning (visible by default).
>
> As part of this though, I'd suggest amending the documentation for
> DeprecationWarning [1] to specifically cover how to turn it off
> programmatically (`warnings.simplefilter("ignore",
> DeprecationWarning)`), at the command line (`python -W
> ignore::DeprecationWarning ...`), and via the environment
> (`PYTHONWARNINGS=ignore::DeprecationWarning`).
>
> (Structurally, I'd probably put that at the end of the warnings
> listing as a short introduction to warnings management, with links out
> to the relevant sections of the documentation, and just use
> DeprecationWarning as the specific example)
>
> Cheers,
> Nick.
>
> [1] https://docs.python.org/3/library/exceptions.html#DeprecationWarning
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> gjcarneiro%40gmail.com
>



-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Gustavo Carneiro]

On 12 October 2017 at 11:20, Steve Holden  wrote:

> On Thu, Oct 12, 2017 at 9:20 AM, Mike Miller 
> wrote:
>
>>
>> On 2017-10-12 00:36, Stéfane Fermigier wrote:
>>
>>> "An object that is not defined by its attributes, but rather by a thread
>>> of continuity and its identity." (from https://en.wikipedia.org/wiki/
>>> Domain-driven_design#Building_blocks)
>>>
>>
>> Not sure I follow all this, but Python objects do have identities once
>> instantiated.  e.g. >>> id('')
>>
>> ​It seems to me that the quoted document is attempting to make a
> distinction ​similar to the one between classes (entities) and instances
> (value objects). The reason I liked "row" as a name is because it
> resembles  "vector" and hence is loosely assocaited with the concept of a
> tuple as well as being familiar to database users. In fact the answer to a
> relational query was, I believe, originally formally defined as a set of
> tuples.
>

But rows and tuples are usually immutable, at least in database terms.
These data classes are not immutable (by default).  If you want tuple-like
behaviour, you can continue to use tuples.

I see dataclasses as something closer to C `struct`.  Most likely someone
already considered `struct` as name; if not, please consider it.  Else
stick with dataclass, it's a good name IMHO.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Gustavo Carneiro]

Hi, it is not clear whether anything is done to total_cost:

def total_cost(self) -> float:

Does this become a property automatically, or is it still a method call?
To that end, some examples of *using* a data class, not just defining one,
would be helpful.

If it remains a normal method, why put it in this example at all? Makes
little sense...

Otherwise I really like this idea, thanks!


On 8 September 2017 at 15:57, Eric V. Smith  wrote:

> I've written a PEP for what might be thought of as "mutable namedtuples
> with defaults, but not inheriting tuple's behavior" (a mouthful, but it
> sounded simpler when I first thought of it). It's heavily influenced by the
> attrs project. It uses PEP 526 type annotations to define fields. From the
> overview section:
>
> @dataclass
> class InventoryItem:
> name: str
> unit_price: float
> quantity_on_hand: int = 0
>
> def total_cost(self) -> float:
> return self.unit_price * self.quantity_on_hand
>
> Will automatically add these methods:
>
>   def __init__(self, name: str, unit_price: float, quantity_on_hand: int =
> 0) -> None:
>   self.name = name
>   self.unit_price = unit_price
>   self.quantity_on_hand = quantity_on_hand
>   def __repr__(self):
>   return f'InventoryItem(name={self.name!r},unit_price={self.unit_pri
> ce!r},quantity_on_hand={self.quantity_on_hand!r})'
>   def __eq__(self, other):
>   if other.__class__ is self.__class__:
>   return (self.name, self.unit_price, self.quantity_on_hand) == (
> other.name, other.unit_price, other.quantity_on_hand)
>   return NotImplemented
>   def __ne__(self, other):
>   if other.__class__ is self.__class__:
>   return (self.name, self.unit_price, self.quantity_on_hand) != (
> other.name, other.unit_price, other.quantity_on_hand)
>   return NotImplemented
>   def __lt__(self, other):
>   if other.__class__ is self.__class__:
>   return (self.name, self.unit_price, self.quantity_on_hand) < (
> other.name, other.unit_price, other.quantity_on_hand)
>   return NotImplemented
>   def __le__(self, other):
>   if other.__class__ is self.__class__:
>   return (self.name, self.unit_price, self.quantity_on_hand) <= (
> other.name, other.unit_price, other.quantity_on_hand)
>   return NotImplemented
>   def __gt__(self, other):
>   if other.__class__ is self.__class__:
>   return (self.name, self.unit_price, self.quantity_on_hand) > (
> other.name, other.unit_price, other.quantity_on_hand)
>   return NotImplemented
>   def __ge__(self, other):
>   if other.__class__ is self.__class__:
>   return (self.name, self.unit_price, self.quantity_on_hand) >= (
> other.name, other.unit_price, other.quantity_on_hand)
>   return NotImplemented
>
> Data Classes saves you from writing and maintaining these functions.
>
> The PEP is largely complete, but could use some filling out in places.
> Comments welcome!
>
> Eric.
>
> P.S. I wrote this PEP when I was in my happy place.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/gjcarneir
> o%40gmail.com
>



-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Someons's put a "Python 2.8" on GitHub

[Gustavo Carneiro]

On 10 December 2016 at 13:49, M.-A. Lemburg  wrote:
[...]

> Regardless of the name, it'll be interesting to see whether
> there's demand for such a fork. Without a website, binaries
> to download, documentation, etc. it's still in the very early
> stages.
>

IMHO, whether or not there is demand for this release should be
irrelevant.  Caving in to Python 2.8 demand is trading off some short term
gains (adding some Python 3 features to code bases locked into Python 2),
in detriment of a big long-term risk, which is that the Python language
permanently forks into two versions: Python 2 and Python 3.

Right now we have a solid expectation that eventually Python 2 is going to
be legacy and most code bases will convert to Python 3.  If we somehow
endorse Python 2.8, many developers will be tempted to just stick with
Python 2 forever.  This would be very very bad for the future of the
language as whole.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Gustavo Carneiro]

On 15 April 2016 at 18:03, Victor Stinner  wrote:

> Hum.
>
> if (width == 0
> and height == 0
> and color == 'red'
> and emphasis == 'strong'
> or highlight > 100):
> raise ValueError("sorry, you lose")
>
> Please remove one space to vertically align "and" operators with the
> opening parenthesis:
>
> if (width == 0
>and height == 0
>and color == 'red'
>and emphasis == 'strong'
>or highlight > 100):
> raise ValueError("sorry, you lose")
>

Personally, I think what you propose looks ugly.  The first version looks
so much better.


It helps to visually see that the multiline test and the raise
> instruction are in two different blocks.


The only thing I would add would be an empty line to help distinguish the
if expression block from the "then" code block:

if (width == 0
and height == 0
and color == 'red'
and emphasis == 'strong'
or highlight > 100):

raise ValueError("sorry, you lose")


-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncio: how to interrupt an async def w/ finally: ( e.g. Condition.wait() )

[Gustavo Carneiro]

I tried to reproduce the problem you describe, but failed.  Here's my test
program (forgive the awful tab indentation, long story):

--
import asyncio

async def foo():
print("resource acquire")
try:
await asyncio.sleep(100)
finally:
print("resource release")


async def main():
task = asyncio.ensure_future(foo())
print("task created")
await asyncio.sleep(0)
print("about to cancel task")
task.cancel()
print("task cancelled, about to wait for it")
try:
await task
except asyncio.CancelledError:
pass
print("waited for cancelled task")


if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
---

I get this output:


10:54:28 ~/Documents$ python3.5 foo.py
task created
resource acquire
about to cancel task
task cancelled, about to wait for it
resource release
waited for cancelled task


Which seems to indicate that the finally clause is correctly executed when
the task is waited for, after being cancelled.

But maybe I completely misunderstood your problem...


On 19 December 2015 at 21:40, Matthias Urlichs  wrote:

> On 19.12.2015 20:25, Guido van Rossum wrote:
> > Perhaps you can add a check for a simple boolean 'stop' flag to your
> > condition check, and when you want to stop the loop you set that flag
> > and then call notify() on the condition. Then you can follow the
> > standard condition variable protocol instead of all this nonsense. :-)
> Your example does not work.
>
> >def stop_it(self):
> >self.stopped = True
> >self.uptodate.notify()
>
> self.uptodate needs to be locked before I can call .notify() on it.
> Creating a new task just for that seems like overkill, and I'd have to
> add a generation counter to prevent a race condition. Doable, but ugly.
>
> However, this doesn't fix the generic problem; Condition.wait() was just
> what bit me today.
> When a non-async generator goes out of scope, its finally: blocks will
> execute. An async procedure call whose refcount reaches zero without
> completing simply goes away; finally: blocks are *not* called and there
> is *no* warning.
> I consider that to be a bug.
>
> --
> -- Matthias Urlichs
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
>



-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make stacklevel=2 by default in warnings.warn()

[Gustavo Carneiro]

On 20 September 2015 at 07:55, Nathaniel Smith  wrote:

> On Sat, Sep 19, 2015 at 11:44 PM, Serhiy Storchaka 
> wrote:
> > For now the default value of the stacklevel parameter in warnings.warn()
> is
> > 1. But in most cases stacklevel=2 is required, sometimes >2, and I don't
> > know cases that need stacklevel=1. I propose to make the default value of
> > stacklevel to be 2. I think that unlikely this will break existing code.
> But
> > rather can fix existing bugs. If stacklevel=1 is required (I don't know
> > cases), it can be explicitly specified.
>
> +1
>
> I don't have enough fingers to count how many times I've had to
> explain how stacklevel= works to maintainers of widely-used packages
> -- they had no idea that this was even a thing they were getting
> wrong.
>
> OTOH I guess if there is anyone out there who's intentionally using
> stacklevel=1 they might be reasonably surprised at this change. I
> guess for some kinds of warnings stacklevel=2 is not obviously correct
> -- the one that comes to mind is "warning: the computer on the other
> end of this network connection did something weird, continuing
> anyway". OTOOH in this case I'm not sure stacklevel=1 is any better
> either, since the thing being warned about has nothing to do with the
> current call stack at all.
>

In this case you should use the logging module instead.

-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Gustavo Carneiro]

On 5 September 2015 at 09:44, haypo s  wrote:

> 2015-09-05 5:01 GMT+02:00 Guido van Rossum :
> > And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
> > holiday in the US) in case anything unforeseen comes up, but this is
> really
> > the Last Call for this PEP.
>
> String concatenation is inefficient in Python because strings are
> immutable. There is a micro-optimization which tried to reduce the bad
> performances of a+b, but it's better to avoid it.
>
> Python replaces >'abc' 'def'< with a single string >'abcdef'<. It's
> done by the parser, there is no overhead at runtime.
>
> PEP 498 allows to write >'abc' f'string'< which is replaced with
> >'abc' 'string'.__format__()< whereas str+str is a bad practice.
>
> I would prefer to force users to write an explicit '+' to remind them
> that there are more efficient ways to concatenate strings like
> ''.join((str1, str2)) or putting the first string in the f-string:
> >f'abcstring'<.
>
> Section in the PEP:
> https://www.python.org/dev/pepsstring/pep-0498/#concatenating-strings
>
>
That's a very good point!


> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
>



-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Gustavo Carneiro]

Why not allow string concatenation without plus sign only if/when the
implementation becomes optimised to allow compile time concatenation?  This
makes it crystal clear whether the concatenation is compile time or
runtime. If we allow it now, it's hard to tell without looking at the
CPython source code...

On Sat, 5 Sep 2015 16:57 Guido van Rossum  wrote:

On Sat, Sep 5, 2015 at 1:44 AM, haypo s  wrote:

2015-09-05 5:01 GMT+02:00 Guido van Rossum :
> And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
> holiday in the US) in case anything unforeseen comes up, but this is
really
> the Last Call for this PEP.

String concatenation is inefficient in Python because strings are
immutable. There is a micro-optimization which tried to reduce the bad
performances of a+b, but it's better to avoid it.

Python replaces >'abc' 'def'< with a single string >'abcdef'<. It's
done by the parser, there is no overhead at runtime.

PEP 498 allows to write >'abc' f'string'< which is replaced with
>'abc' 'string'.__format__()< whereas str+str is a bad practice.

I would prefer to force users to write an explicit '+' to remind them
that there are more efficient ways to concatenate strings like
''.join((str1, str2)) or putting the first string in the f-string:
>f'abcstring'<.

Section in the PEP:
https
://

www.python.org
/dev/

pepsstring

/pep-0498/#concatenating-strings


Victor

What does the extra + buy users? It's just more punctuation noise. The
implementation is intentionally left unspecified by the PEP; it should be
possible to design an implementation that combines all the parts together
more efficiently than the use of + (e.g. using a "string builder" object
internally).


-- 

--Guido van Rossum (python.org /~
guido )

___
Python-Dev mailing list
Python-Dev@python.org
https ://
mail.python.org
/mailman/
listinfo
/
python-dev

Unsubscribe: https

://

mail.python.org

/mailman/options/

python-dev
/

gjcarneiro%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] provisional status for asyncio

[Gustavo Carneiro]

I think this is a packaging problem.

In an ideal world, Python would ship some version of asyncio, but you would
also be able to pip install a newer version into your virtual environment,
and it would override the version in stdlib.

As it stands now, however, if you pip install another version of asyncio,
the version in stdlib still takes precedence.  What I end up doing in my
(non open source) projects is to include a copy of asyncio and manually
modify sys.path to point to it.

Can we fix pip/virtualenv instead?
On 28 Aug 2015 10:02 am, Victor Stinner victor.stin...@gmail.com wrote:

 Hi,

 2015-08-27 23:15 GMT+02:00 Yury Selivanov yselivanov...@gmail.com:
  Recently, in an asyncio related issue [1], Guido said that new features
  for asyncio have to wait till 3.6, since asyncio is no longer a
 provisional
  package. (...)
  For example, there is an issue [2] to add starttls support to asyncio.
  (...)
  Aside from adding new APIs, we also have to improve debugging
  capabilities.
  (...)

 I would propose something more radical: remove asyncio from the stdlib.

 PEP 411: While it is considered an unlikely outcome, such packages
 *may even be removed* from the standard library without a deprecation
 period if the concerns regarding their API or maintenance prove
 well-founded.

 As an asyncio developer, I'm not sure that asyncio fits well into the
 Python stdlib. The release cycle of feature release is long (18
 months? or more?), the release cycle for bugfix release is sometimes
 also too long (1 month at least). It's also frustrating to have subtle
 API differences between Python 3.3, 3.4 and 3.5. For example,
 asyncio.JoinableQueue was removed in Python 3.5, asyncio.Queue can
 *now* be used instead, but asyncio.JoinableQueue should be used on
 older Python version... It means that you have to write different code
 depending on your Python version to support all Python versions.

 I can give much more examples of missing asyncio features. Example:
 Windows proactor event loop doesn't support signals (CTRL+C) nor UDP.

 asyncio is moving so fast, that changes are not documented at all in
 Misc/NEWS or Doc/whatsnews/x.y.rst. I tried to document changes in my
 fork Trollius. See its changelog to have an idea how fast asyncio is
 still changing:
 http://trollius.readthedocs.org/changelog.html

 I don't know how users will react to the removal of asyncio from the
 stdlib (asyncio is not trusted/supported by Python?).

 The idea is to install it using pip: pip install asyncio. The major
 difference is that pip install -U asyncio allows to retrieve the
 latest asyncio version, independently of your Python version. Hum, I
 don't know if it works with Python 3.4 (which asyncio module is used
 in this case?).

 Developing asyncio only on Github would avoid the risk of having
 patches temporary only in Github or only in CPython. It avoids the
 need to synchronize the Git (Github) and Mercurial (CPython)
 repositories.


 Compare Python release dates with Twisted, Tornado and eventlet release
 dates.

 Twisted releases:

 * 2015-01-30: 15.0.0
 * 2015-04-13: 15.1.0
 * 2015-05-19: 15.2.0
 * 2015-08-04: 15.3.0

 Tornado releases:

 * 2014-07-15: 4.0
 * 2015-02-07: 4.1
 * 2015-05-27: 4.2
 * 2015-07-17: 4.2.1

 eventlet releases:

 * 2015-02-25: 0.17.1
 * 2015-04-03: 0.17.2
 * 2015-04-09: 0.17.3
 * 2015-05-08: 0.17.4

 Victor
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] provisional status for asyncio

[Gustavo Carneiro]

On 28 August 2015 at 17:11, Yury Selivanov yselivanov...@gmail.com wrote:

 On 2015-08-28 11:44 AM, Brett Cannon wrote:


 Unfortunately, separating it from the standard library is something
 that I don't think we can do so late in the 3.5 release candidates
 process.


 Ultimately it's Larry's call, but I don't see why we couldn't. If we were
 talking about something as low-level as the urllib package then I would
 agree, but beyond its own tests is there anything in the stdlib that
 depends on asyncio?


 As Victor already replied, asyncio is pretty self contained, and
 nothing in the stdlib depends on it.

 If we really can remove it, let's consider it.


-1 for removing it.  It is way too late for 3.5 IMHO.  You should have
proposed it at least 6 months ago.  This feels too rushed.

+1 for changing sys.path to allow pip-installed asyncio to override the
stdlib version (for 3.6).  Some developers complain that NodeJS's packaging
system is better that Python, maybe this lack of flexibility is one of the
reasons...


-- 
Gustavo J. A. M. Carneiro
Gambit Research
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Gustavo Carneiro]

On 1 May 2015 at 16:31, Guido van Rossum gu...@python.org wrote:

 On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel stefan...@behnel.de wrote:

 Yury Selivanov schrieb am 30.04.2015 um 03:30:
  1. Terminology:
  - *native coroutine* term is used for async def functions.

 When I read native, I think of native (binary) code. So native
 coroutine sounds like it's implemented in some compiled low-level
 language. That might be the case (certainly in the CPython world), but
 it's
 not related to this PEP nor its set of examples.


  We should discuss how we will name new 'async def' coroutines in
  Python Documentation if the PEP is accepted.

 Well, it doesn't hurt to avoid obvious misleading terminology upfront.


 I think obvious[ly] misleading is too strong, nobody is trying to
 mislead anybody, we just have different associations with the same word.
 Given the feedback I'd call native coroutine suboptimal (even though I
 proposed it myself) and I am now in favor of using async function.


But what if you have async methods?  I know, a method is almost a function,
but still, sounds slightly confusing.

IMHO, these are really classical coroutines.  If gevent calls them
coroutines, I don't think asyncio has any less right to call them
coroutines.

You have to ask yourself this: a new programmer, when he sees mentions of
coroutines, how likely is he to understand what he is dealing with?  What
about async function?  The former is a well known concept, since decades
ago, while the latter is something he probably (at least me) never heard of
before.

For me, an async function is just as likely to be an API that is
asynchronous in the sense that it takes an extra callback parameter to be
called when the asynchronous work is done.

I think coroutine is the name of a concept, not a specific implementation.

Cheers,

-- 
Gustavo J. A. M. Carneiro
Gambit Research
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Gustavo Carneiro]

On 21 April 2015 at 11:56, Rob Cliffe rob.cli...@btinternet.com wrote:

  On 21/04/2015 10:33, Cory Benfield wrote:

 On 21 April 2015 at 10:10, Chris Angelico ros...@gmail.com 
 ros...@gmail.com wrote:

  At this point, you may want to just stop caring about the exact type.
 Part of the point of gradual typing is that you can short-cut a lot of
 this. And quite frankly, this isn't really helping anything. Just skip
 it and say that it's Union[Mapping, Iterable, None].

  I think this is my problem with the proposal. This change doesn't
 catch any of the bugs anyone was going to hit (no-one is passing a
 callable to this parameter), and it doesn't catch any of the bugs
 someone might actually hit (getting the tuple elements in the wrong
 order, for example). At this point the type signature is worse than
 useless.

  Exactly.  At this point putting the type signatures in seems to be a
 pointless exercise in masochism (for the author) and sadism (for the
 reader).

 The refreshing thing about Python is that it is a fantastic, *concise*,
 dynamically typed language, at the opposite end of the spectrum from C++
 (ugh!).
 We have got used to the consequences (good and bad) of this:
 Duck typing, e.g. a function to sort numbers (sorry Tim Peters bad
 example) turns out to support any kind of object (e.g. strings) that
 supports comparison.
 Not to mention objects of some class that will be written in 5
 years time.
 (Adding a type hint that restricted the argument to say a sequence
 of numbers turns out to be a mistake.  And what is a number?
  Is Fraction?  What about complex numbers, which can't be sorted?
 What if the function were written before the Decimal class?)
 Errors are often not caught until run time that would be caught at
 compile time in other languages (though static code checkers help).
 (Not much of a disadvantage because of Python's superb error
 diagnostics.)
  Python code typically says what it is doing, with the minimum of
 syntactic guff.  (Well, apart from colons after if/while/try etc. :-) )
 Which makes it easy to read.
 Now it seems as if this proposal wants to start turning Python in the C++
 direction, encouraging adding ugly boilerplate code.  (This may only be
 tangentially relevant, but I want to scream when I see some combination of
 public/private/protected/static/extern etc., most of which I don't
 understand.)

 Chris A makes the valid point (if I understand correctly) that
 Authors of libraries should make it as easy as possible to
   (i) know what object types can be passed to functions
   (ii) diagnose when the wrong type of object is passed
 Authors of apps are not under such obligation, they can basically do
 what they want.

 Well,
 (i) can be done with good documentation (docstrings etc.).


Documentation is not checked.  It often loses sync with the actual code.
Docs say one thing, code does another.

Documenting types in docstrings forces you to repeat yourself.  You have to
write the parameter names twice, once in the function definition line,
another in the docstring.  And you need to understand the syntax of
whatever docstring format will be used to check the code.

I'm sure I'm not the only one that thinks that type hints actually *improve
readability*.  I will not argue whether it makes the code uglier or
prettier, but I am certain that most of the time the type hints make the
code easier to read because you don't have to spend so much mental effort
trying to figure out gee, I wonder if this parameter is supposed to be
bool, string, or int?  The type (or types) that a parameter is expected to
have becomes explicit, otherwise you have to read the entire function body
to understand.


 (ii) can be done with appropriate runtime checks and good error
 messages.
 E.g. (Cory's example) I'm sure it is possible to test somehow if an object
 is file-like (if only by trying to access it like a file).
 Is thorough argument checking and provision of good diagnostics going to
 be easy for the library author?  No.  Is it going to be a lot of work to do
 thoroughly?  Almost certainly yes.
 But what the hell, writing a production-quality library is not an exercise
 for newbies.

 It seems to me that type hints are attempting to be a silver bullet and to
 capture in a simple formula what is often, in practice, *not simple at
 all*, viz. Is this passed object suitable?.  Attempting - and failing,
 except in the simplest cases.


Even if it only helps in the simplest cases, it saves the reader of the
code a lot of effort if you add up all those little cases together.

My suggestion, wherever an annotated type is too complex, don't annotate
it.  If you do /only/ the simple cases, the end result is easier to read.



 Apologies, Guido, but:
 There was once a Chinese student who was a marvellous painter.  He painted
 a perfect life-like picture of a snake.
 But he was unable to stop and leave it alone.  

Re: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions

[Gustavo Carneiro]

On 12 March 2014 15:21, Brett Cannon br...@python.org wrote:




 On Wed, Mar 12, 2014 at 11:16 AM, Chris Angelico ros...@gmail.com wrote:

 On Thu, Mar 13, 2014 at 2:14 AM, Brett Cannon br...@python.org wrote:
  While I'm +0 on the idea, I'm -1 on the syntax;



 I just don't like having a
  colon in an expression.


lambda: is an expression and has a colon in it.

It would be weird to use except: without a colon IMHO, because we are
already used to write except: with colon everywhere else...

Which is why there are alternatives listed, and the best four of them
 (including the proposed one) ranked.


 `value = (expr except Exception then default)` would be fine with me.


Do we really need to introduce a new keyword 'then' ?

I'm +1 on the original proposal (expr except Exception: default)

-- 
Gustavo J. A. M. Carneiro
Gambit Research
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

[Gustavo Carneiro]

On 17 February 2014 11:14, M.-A. Lemburg m...@egenix.com wrote:

 On 15.02.2014 07:03, Stephen J. Turnbull wrote:
  M.-A. Lemburg writes:
 
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.
 
  I don't understand this statement.  The theory is that they *should*
  fail.
 
  The example of sort is a good one.  Sometimes you want missing values
  to be collected at the beginning of a list, sometimes at the end.
  Sometimes you want them treated as top elements, sometimes as bottom.
  And sometimes it is a real error for missing values to be present.
  Not to mention that sometimes the programmer simply hasn't thought
  about the appropriate policy.  I don't think Python should silently
  impose a policy in that case, especially given that the programmer may
  have experience with any of the above treatments in other contexts.

 None is special in Python and has always (and intentionally) sorted
 before any other object. In data processing and elsewhere in Python
 programming, it's used to signal: no value available.

 Python 3 breaks this notion by always raising an exception when
 using None in an ordered comparison, making it pretty much useless
 for the above purpose.

 Yes, there are ways around this, but none of them are intuitive.

 Here's a particularly nasty case:

  l = [(1, None), (2, None)]
  l.sort()
  l
 [(1, None), (2, None)]

  l = [(1, None), (2, None), (3, 4)]
  l.sort()
  l
 [(1, None), (2, None), (3, 4)]

  l = [(1, None), (2, None), (3, 4), (2, 3)]
  l.sort()
 Traceback (most recent call last):
   File stdin, line 1, in module
 TypeError: unorderable types: int()  NoneType()


Maybe Python 3 should have a couple of None-like objects that compare the
way you want: AlwaysComparesLess and AlwaysComparesGreater, but with better
names (maybe 'PlusInfinity' and 'MinusInfinity'?).  Just leave None alone,
please.




 --
 Marc-Andre Lemburg
 eGenix.com

 Professional Python Services directly from the Source  (#1, Feb 17 2014)
  Python Projects, Consulting and Support ...   http://www.egenix.com/
  mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
  mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
 
 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

 : Try our mxODBC.Connect Python Database Interface for free ! ::

eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com




-- 
Gustavo J. A. M. Carneiro
Gambit Research LLC
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

[Gustavo Carneiro]

On 17 February 2014 11:43, M.-A. Lemburg m...@egenix.com wrote:

 On 17.02.2014 12:23, Gustavo Carneiro wrote:
  On 17 February 2014 11:14, M.-A. Lemburg m...@egenix.com wrote:
 
  On 15.02.2014 07:03, Stephen J. Turnbull wrote:
  M.-A. Lemburg writes:
 
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.
 
  I don't understand this statement.  The theory is that they *should*
  fail.
 
  The example of sort is a good one.  Sometimes you want missing values
  to be collected at the beginning of a list, sometimes at the end.
  Sometimes you want them treated as top elements, sometimes as bottom.
  And sometimes it is a real error for missing values to be present.
  Not to mention that sometimes the programmer simply hasn't thought
  about the appropriate policy.  I don't think Python should silently
  impose a policy in that case, especially given that the programmer may
  have experience with any of the above treatments in other contexts.
 
  None is special in Python and has always (and intentionally) sorted
  before any other object. In data processing and elsewhere in Python
  programming, it's used to signal: no value available.
 
  Python 3 breaks this notion by always raising an exception when
  using None in an ordered comparison, making it pretty much useless
  for the above purpose.
 
  Yes, there are ways around this, but none of them are intuitive.
 
  Here's a particularly nasty case:
 
  l = [(1, None), (2, None)]
  l.sort()
  l
  [(1, None), (2, None)]
 
  l = [(1, None), (2, None), (3, 4)]
  l.sort()
  l
  [(1, None), (2, None), (3, 4)]
 
  l = [(1, None), (2, None), (3, 4), (2, 3)]
  l.sort()
  Traceback (most recent call last):
File stdin, line 1, in module
  TypeError: unorderable types: int()  NoneType()
 
 
  Maybe Python 3 should have a couple of None-like objects that compare the
  way you want: AlwaysComparesLess and AlwaysComparesGreater, but with
 better
  names (maybe 'PlusInfinity' and 'MinusInfinity'?).  Just leave None
 alone,
  please.

 This doesn't only apply to numeric comparisons. In Python 2 you
 can compare None with any kind of object and it always sorts first,
 based on the intuition that nothing is less than anything :-)

 I still think that relying on your intuition is not the right way for
Python.  Refuse the temptation to guess.


 FWIW, I don't think we need to invent a new name for it, just add
 an appropriate tp_richcompare slot to the PyNoneType or readd the
 special case to Object/object.c. This would also aid in porting
 existing Python 2 code to Python 3.


Based on your comment, SortsFirst and SortsLast sound like good names ;-)

These would be universal sortable objects, that could be compared to any
other type.




 --
 Marc-Andre Lemburg
 eGenix.com

 Professional Python Services directly from the Source  (#1, Feb 17 2014)
  Python Projects, Consulting and Support ...   http://www.egenix.com/
  mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
  mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
 
 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

 : Try our mxODBC.Connect Python Database Interface for free ! ::

eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/




-- 
Gustavo J. A. M. Carneiro
Gambit Research LLC
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

[Gustavo Carneiro]

On 17 February 2014 12:30, M.-A. Lemburg m...@egenix.com wrote:

 On 17.02.2014 13:19, Serhiy Storchaka wrote:
  17.02.14 13:56, M.-A. Lemburg написав(ла):
  Yes, but that's not the point. Unlike strings or other mixed types that
  you cannot compare, None is used as placeholder in data processing as
  special value to mean no value available.
 
  Isn't float('nan') such placeholder?

 You can easily construct other such placeholders, but None was intended
 for this purpose:

 http://docs.python.org/2.7/c-api/none.html?highlight=none#Py_None

 
 The Python None object, denoting lack of value. ...
 

  You intentionally use such values in programming. It's not a bug to
  have None in a data list or as value of a variable.
 
  You can't have None in array('f'), you can't add or multiply by None.
 Relation operators don't looks
  an exception here. Applying sorted() to a list which contains numbers
 and Nones makes as much sense
  as applying sum() to it.

 Of course, you cannot apply any operations with None - it doesn't
 have a value -, but you can compare it to other objects and it provides
 a consistent behavior in Python 2. Python 3 is missing such an object.


I agree with you that Python 3 could use such an object.  Just don't make
it the default.  Leave None as it is.

Also I agree that my previous naming suggestions are bad.  What we need is
only one (or two) additional object whose main semantic meaning is still
no value, but which also adds a meaning of comparable.  Then it's a
matter of choosing a good name for it, with lots of bikeshedding involved.
 Just lets not change None only because we're too lazy to discuss a proper
alternative name.  Also this use case is not _that_ common, so it's ok if
it has a slightly longer name than None.

Also think of the implications of changing None at this point.  It would
allow us to write programs that work Python = 3.5 and Python = 2.7, but
fail mysteriously in all other versions in between.  What a mess that would
be...

-- 
Gustavo J. A. M. Carneiro
Gambit Research LLC
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] News from asyncio

[Gustavo Carneiro]

On 27 January 2014 10:55, Victor Stinner victor.stin...@gmail.com wrote:

 2014-01-27 Antoine Pitrou solip...@pitrou.net:
  On Mon, 27 Jan 2014 10:45:37 +0100
  Victor Stinner victor.stin...@gmail.com wrote:
 
  - Tulip #111: StreamReader.readexactly() now raises an
  IncompleteReadError if the
  end of stream is reached before we received enough bytes, instead of
 returning
  less bytes than requested.
 
  Why not simply EOFError?

 IncompleteReadError has two additionnal attributes:

 - partial: incomplete received bytes
 - expected: total number of expected bytes (n parameter of readexactly)

 I prefer to use a different exception to ensure that these attributes
 are present. I don't like having to check hasattr(exc, ...).

 Before this change, readexactly(n) returned the partial received bytes
 if the end of the stream was reached.


I had the same doubt.  Note also that IncompleteReadError is a subclass of
EOFError, so you can catch EOFError if you like.



 Victor
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com




-- 
Gustavo J. A. M. Carneiro
Gambit Research LLC
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 as a Default in Linux Distros

[Gustavo Carneiro]

On Wed, Jul 24, 2013 at 6:04 PM, Terry Reedy tjre...@udel.edu wrote:

 On 7/24/2013 5:12 AM, Bohuslav Kabrda wrote:

 Hi all, in recent days, there has been a discussion on fedora-devel
 (see thread [1]) about moving to Python 3 as a default.


 Default-shift is a known natural language phenomenon.
 https://en.wikipedia.org/wiki/**Retronymhttps://en.wikipedia.org/wiki/Retronym
 It is inevitably messy in the middle. Even worse for us, Python' is both
 a natural and artificial language term and the default meaning will not
 necessarily be synchronized across the two domains.

 Not being a current *nix user, I have no opinion on the specific
 alternatives. However, I am concerned about newcomers being mis-served by
 being mislead into thinking that 'Python' is python 2 rather python 3. We
 see this in some newbie posts on Python list. (This is especially a concern
 when default Python is still, say, 2.5.)  I have no idea, though, how to
 avoid breaking scripts while directing newcomers in the proper direction.


+1

Maybe we could make interactive python2 scripts print a big warning
advising users to use python 3 instead of 2, if possible?  I mean, if
stdout is a TTY.  This way, even if 'python' is python2, newbies will see
the warning.  I think every developer (newbie or not) sooner or later will
try a python interactive shell, and so will see the warning.  It is
important also for experienced python developers to be told explicitly that
they should be thinking about switching to python 3, at some point.





 --
 Terry Jan Reedy


 __**_
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/**mailman/listinfo/python-devhttp://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
 gjcarneiro%40gmail.comhttp://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com




-- 
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] stat module in C -- what to do with stat.py?

[Gustavo Carneiro]

On Fri, Jun 21, 2013 at 8:20 PM, Steven D'Aprano st...@pearwood.infowrote:

 On 21/06/13 01:35, Benjamin Peterson wrote:

 2013/6/20 Charles-François Natali cf.nat...@gmail.com:

 2013/6/20 Thomas Wouters tho...@python.org:

 If the .py file is going to be wrong or incomplete, why would we want to
 keep it -- or use it as fallback -- at all? If we're dead set on having
 a
 .py file instead of requiring it to be part of the interpreter
 (whichever
 that is, however it was built), it should be generated as part of the
 build
 process. Personally, I don't see the value in it; other implementations
 will
 need to do *something* special to use it anyway.


 That's not correct. Other implementations can do exactly what CPython 3.3
 does, namely just use stat.py as given. Not all implementations necessarily
 care about multiple platforms where stat constants are likely to change.



  That's exactly my rationale for pushing for removal.


 +1 to nixing it.


 -1

 Reading the Python source code is a very good way for beginner programmers
 to learn about things like this.


On the other hand, it is counter-productive to learn about code that is
conceptually _wrong_.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - request for pronouncement

[Gustavo Carneiro]

On Fri, May 31, 2013 at 11:34 AM, Łukasz Langa luk...@langa.pl wrote:

 On 31 maj 2013, at 12:18, Gustavo Carneiro gjcarne...@gmail.com wrote:

  It is not clear from the PEP (up until the end of the User API section
 at least) when, if ever, is this implementation of fun ever called.  I
 mean, what type of 'arg' triggers a dispatch to this function body?

 I added a sentence clarifying that. See the commit:
 http://hg.python.org/peps/rev/4d6c827944c4

 Does that address your concern?


Yes, much better now.  Thank you!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - request for pronouncement

[Gustavo Carneiro]

Sorry, maybe I am too late to comment on this, but,

   @singledispatch
  ... def fun(arg, verbose=False):
  ... if verbose:
  ... print(Let me just say,, end= )
  ... print(arg)

It is not clear from the PEP (up until the end of the User API section at
least) when, if ever, is this implementation of fun ever called.  I mean,
what type of 'arg' triggers a dispatch to this function body?
I am guessing that when the arg does not match the type of any of the other
registered functions, this function body is used by default.  But it is
only a guess, the PEP doesn't state this clearly.

If my guess is true, would it be reasonable to update the example def fun
code to reflect this, e.g., to print(Warning: I do not know what to do
with arg {} of type {}.format(arg, type(arg)).

So my comment is just about clarity of the PEP text.  I do not wish to
interfere with pronouncement.

Thanks.



On Fri, May 31, 2013 at 10:46 AM, Łukasz Langa luk...@langa.pl wrote:

 Hello python-dev,

 PEP 443 is ready for final review. I'm attaching the latest
 version below for convenience. The full history of changes
 is available here: http://hg.python.org/peps/log/tip/pep-0443.txt

 A reference implementation for PEP 443 is available at:
 http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l363

 with relevant tests here:

 http://hg.python.org/features/pep-443/file/tip/Lib/test/test_functools.py#l855

 and documentation here:

 http://hg.python.org/features/pep-443/file/tip/Doc/library/functools.rst#l189

 There's also an official backport for 2.6 - 3.3 already up:
 https://pypi.python.org/pypi/singledispatch



 PEP: 443
 Title: Single-dispatch generic functions
 Version: $Revision$
 Last-Modified: $Date$
 Author: Łukasz Langa luk...@langa.pl
 Discussions-To: Python-Dev python-dev@python.org
 Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 22-May-2013
 Post-History: 22-May-2013, 25-May-2013, 31-May-2013
 Replaces: 245, 246, 3124


 Abstract
 

 This PEP proposes a new mechanism in the ``functools`` standard library
 module that provides a simple form of generic programming known as
 single-dispatch generic functions.

 A **generic function** is composed of multiple functions implementing
 the same operation for different types. Which implementation should be
 used during a call is determined by the dispatch algorithm. When the
 implementation is chosen based on the type of a single argument, this is
 known as **single dispatch**.


 Rationale and Goals
 ===

 Python has always provided a variety of built-in and standard-library
 generic functions, such as ``len()``, ``iter()``, ``pprint.pprint()``,
 ``copy.copy()``, and most of the functions in the ``operator`` module.
 However, it currently:

 1. does not have a simple or straightforward way for developers to
create new generic functions,

 2. does not have a standard way for methods to be added to existing
generic functions (i.e., some are added using registration
functions, others require defining ``__special__`` methods, possibly
by monkeypatching).

 In addition, it is currently a common anti-pattern for Python code to
 inspect the types of received arguments, in order to decide what to do
 with the objects. For example, code may wish to accept either an object
 of some type, or a sequence of objects of that type.

 Currently, the obvious way to do this is by type inspection, but this
 is brittle and closed to extension. Abstract Base Classes make it easier
 to discover present behaviour, but don't help adding new behaviour.
 A developer using an already-written library may be unable to change how
 their objects are treated by such code, especially if the objects they
 are using were created by a third party.

 Therefore, this PEP proposes a uniform API to address dynamic
 overloading using decorators.


 User API
 

 To define a generic function, decorate it with the ``@singledispatch``
 decorator. Note that the dispatch happens on the type of the first
 argument, create your function accordingly::

from functools import singledispatch
@singledispatch
   ... def fun(arg, verbose=False):
   ... if verbose:
   ... print(Let me just say,, end= )
   ... print(arg)

 To add overloaded implementations to the function, use the
 ``register()`` attribute of the generic function. It is a decorator,
 taking a type parameter and decorating a function implementing the
 operation for that type::

@fun.register(int)
   ... def _(arg, verbose=False):
   ... if verbose:
   ... print(Strength in numbers, eh?, end= )
   ... print(arg)
   ...
@fun.register(list)
   ... def _(arg, verbose=False):
   ... if verbose:
   ... print(Enumerate this:)
   ... for i, elem in enumerate(arg):
   ... print(i, elem)

 To enable registering lambdas and pre-existing functions, the
 ``register()`` attribute can be used in a functional 

[Python-Dev] OSError.errno = exception hierarchy?

[Gustavo Carneiro]

Apologies if this has already been discussed.

I was expecting that by now, python 3.0, the following code:

# clean the target dir
import errno
try:
shutil.rmtree(trace_output_path)
except OSError, ex:
if ex.errno not in [errno.ENOENT]:
raise

Would have become something simpler, like this:

# clean the target dir
try:
shutil.rmtree(trace_output_path)
except OSErrorNoEntry:   # or maybe os.ErrorNoEntry
pass

Apparently no one has bothered yet to turn OSError + errno into a hierarchy
of OSError subclasses, as it should.  What's the problem, no will to do it,
or no manpower?

Regards,

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OSError.errno = exception hierarchy?

[Gustavo Carneiro]

(cross-posting back to python-dev to finalize discussions)

2009/4/2 Guido van Rossum gu...@python.org
[...]

  The problem you report:
 
   try:
 ...
   except OSWinError:
 ...
   except OSLinError:
 ...
 
 
  Would be solved if both OSWinError and OSLinError were always defined in
  both Linux and Windows Python.  Programs could be written to catch both
  OSWinError and OSLinError, except that on Linux OSWinError would never
  actually be raised, and on Windows OSLinError would never occur.  Problem
  solved.

 Yeah, but now you'd have to generate the list of exceptions (which
 would be enormously long) based on the union of all errno codes in the
 universe.

 Unless you only want to do it for some errno codes and not for others,
 which sounds like asking for trouble.

 Also you need a naming scheme that works for all errnos and doesn't
 require manual work. Frankly, the only scheme that I can think of that
 could be automated would be something like OSError_ENAME.

 And, while OSError is built-in, I think these exceptions (because
 there are so many) should not be built-in, and probably not even live
 in the 'os' namespace -- the best place for them would be the errno
 module, so errno.OSError_ENAME.

  The downsides of this?  I can only see memory, at the moment, but I might
 be
  missing something.

 It's an enormous amount of work to make it happen across all
 platforms. And it doesn't really solve an important problem.


I partially agree.  It will be a lot of work.  I think the problem is valid,
although not very important, I agree.




  Now just one final word why I think this matters.  The currently correct
 way
  to remove a directory tree and only ignore the error it does not exist
 is:
 
  try:
  shutil.rmtree(dirname)
  except OSError, e:
  if errno.errorcode[e.errno] != 'ENOENT':
 raise
 
  However, only very experienced programmers will know to write that
 correct
  code (apparently I am not experienced enought!).

 That doesn't strike me as correct at all, since it doesn't distinguish
 between ENOENT being raised for some file deep down in the tree vs.
 the root not existing. (This could happen if after you did
 os.listdir() some other process deleted some file.)


OK.  Maybe in a generic case this could happen, although I'm sure this won't
happen in my particular scenario.  This is about a build system, and I am
assuming there are no two concurrent builds (or else a lot of other things
would fail anyway).


 A better way might be

 try:
  shutil.rmtree(dir)
 except OSError:
  if os.path.exists(dir):
   raise


Sure, this works, but at the cost of an extra system call.  I think it's
more elegant to check the errno (assuming the corner case you pointed out
above is not an issue).


 Though I don't know what you wish to happen of dir were a dangling
 symlink.

  What I am proposing is that the simpler correct code would be something
  like:
 
  try:
  shutil.rmtree(dirname)
  except OSNoEntryError:
  pass
 
  Much simpler, no?

 And wrong.

  Right now, developers are tempted to write code like:
 
  shutil.rmtree(dirname, ignore_errors=True)
 
  Or:
 
  try:
  shutil.rmtree(dirname)
  except OSError:
  pass
 
  Both of which follow the error hiding anti-pattern [1].
 
  [1] http://en.wikipedia.org/wiki/Error_hiding
 
  Thanks for reading this far.

 Thanks for not wasting any more of my time.


OK, I won't waste more time.  If this were an obvious improvement beyond
doubt to most people, I would pursue it, but since it's not, I can live with
it.

Thanks anyway,

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3142: Add a while clause to generator expressions

[Gustavo Carneiro]

2009/1/19 Vitor Bosshard algor...@yahoo.com





 - Mensaje original 
  De: Gerald Britton gerald.brit...@gmail.com
  Para: Terry Reedy tjre...@udel.edu
  CC: python-dev@python.org
  Enviado: lunes, 19 de enero, 2009 15:03:47
  Asunto: Re: [Python-Dev] PEP 3142: Add a while clause to generator
 expressions
 
  Duly noted and thanks for the feedback!  (just what I was looking for
  actually).  I do disagree with the idea that the proposal, if
  implemented, would make Python harder to learn.  Not sure who would
  find it harder.  Having to find and use takewhile was harder for me.
  I still find that one counter-intuitive.  I would have expected the
  parameters in the reverse order (take something, while something else
  is true).  Tripped me up a few times, which got me thinking about an
  alternative.


 Are you even sure the list comprehension doesn't already shortcut
 evaluation?

 This quick test in 2.6 hints otherwise:


  a = (i for i in range(10) if i**210)
  a.next()
 0
  a.next()
 1
  a.next()
 2
  a.next()
 3
  a.next()
 Traceback (most recent call last):
   File pyshell#32, line 1, in module
 a.next()
 StopIteration


Does not prove anything.

 test.py 
source = iter(xrange(10))
a = (i for i in source if i**210)
print list(a)
print list(source)

 output ---
$ python /tmp/test.py
[0, 1, 2, 3]
[]

While 'a' is being evaluated, the source iterator is first completely
exhausted.  If the source iterator is infinite, an infinite loop is created
and the program doesn't terminate.






  ¡Todo sobre la Liga Mexicana de fútbol! Estadisticas, resultados,
 calendario, fotos y más:
 http://espanol.sports.yahoo.com/
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com




-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python under valgrind

[Gustavo Carneiro]

2008/11/28 Hrvoje Niksic [EMAIL PROTECTED]

 A friend pointed out that running python under valgrind (simply valgrind
 python) produces a lot of invalid read errors.  Reading up on
 Misc/README.valgrind only seems to describe why uninitialized reads should
 occur, not invalid ones.  For example:

 $ valgrind python
 [... lots of output ...]
 ==31428== Invalid read of size 4
 ==31428==at 0x808EBDF: PyObject_Free (in /usr/bin/python2.5)
 ==31428==by 0x810DD0A: (within /usr/bin/python2.5)
 ==31428==by 0x810DD34: PyNode_Free (in /usr/bin/python2.5)
 ==31428==by 0x80EDAD9: PyRun_InteractiveOneFlags (in
 /usr/bin/python2.5)
 ==31428==by 0x80EDDB7: PyRun_InteractiveLoopFlags (in
 /usr/bin/python2.5)
 ==31428==by 0x80EE515: PyRun_AnyFileExFlags (in /usr/bin/python2.5)
 ==31428==by 0x80595E6: Py_Main (in /usr/bin/python2.5)
 ==31428==by 0x8058961: main (in /usr/bin/python2.5)
 ==31428==  Address 0x43bf010 is 3,112 bytes inside a block of size 6,016
 free'd
 ==31428==at 0x4024B4A: free (vg_replace_malloc.c:323)
 ==31428==by 0x8059C07: (within /usr/bin/python2.5)
 ==31428==by 0x80EDAA5: PyRun_InteractiveOneFlags (in
 /usr/bin/python2.5)
 ...

 valgrind claims that Python reads 4 bytes inside a block on which free()
 has already been called.  Is valgrind wrong, or is Python really doing that?
  Googling revealed previous reports of this, normally answered by a
 reference to README.valgrind.  But README.valgrind justifies reading from
 ununitialized memory, which doesn't help me understand how reading from the
 middle of a block of freed memory (more precisely, memory on which the libc
 free() has already been called) would be okay.

 I suppose valgrind could be confused by PyFree's pool address validation
 that intentionally reads the memory just before the allocated block, and
 incorrectly attributes it to a previously allocated (and hence freed) block,
 but I can't prove that.  Has anyone investigated this kind of valgrind
 report?


I can't answer your question directly, but I can tell you that whenever I
have to debug memory problems with python extensions is usually use my own
python compiled with --with-pydebug --without-pymalloc.  It really helps
with valgrind.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problems with the new super()

[Gustavo Carneiro]

2008/5/1 Facundo Batista [EMAIL PROTECTED]:

 2008/5/1, Georg Brandl [EMAIL PROTECTED]:

There may be more implications and surprising behavior surrounding
 this.
 
   I know that the implementation is a compromise, but I'd rather see a
  super()
   whose full semantics can be explained to programmers without using to
   cell variable, f_localsplus and symtable.

 In consideration of what's been said about super() in the past, and
 what is handled here regarding its Py3 implementation, I want to make
 a step  back, and ask:

 Has super() proved more useful than harmful? Which is the value for
 Py3 to keep it?


Since Python supports multiple inheritance, you can't get away from
something like super.  Alternatives for methods chaining to parent classes
are 1. implicit/automatic (like C++, I think), 2. explicit/semi-automatic
(python's super, on one form or another, 3. explicit manual (programmer has
to manually keep track of everything).  Of all these alternatives I prefer
Python's super (in one form or another).  1 is too magic, and 3 is too
error prone and cumbersome.

A better question would be, is multiple inheritance good or bad for
programs? :-)

I think many people's answer to the above would be, generally bad, but there
are exceptions where it helps and can be justified.




 Regards,

 --
 .Facundo

 Blog: http://www.taniquetil.com.ar/plog/
 PyAr: http://www.python.org/ar/
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com




-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Gustavo Carneiro]

On 10/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote:

  Adam  I are now on #python-dev. Can you join?

 I think we successfully resolved this. Adam Olsen will produce a patch
 that allows one to specify a single file descriptor to which a zero
 byte will be written by the C-level signal handler. Twisted and PyGTK
 will have to coordinate about this file descriptor. Glyph believes
 this is possible and sufficient.


Great that a solution was found at last.  But to be honest I don't
understand why my patch (the second one) is not good.  Why is it preferable
for libraries to provide their own file descriptor?

Are people concerned about the overhead of always creating a pipe even it
may not be used?
Always creating a pipe would at least avoid the need for any coordination
between PyGTK and Twisted.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Gustavo Carneiro]

On 11/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote:

 On Dec 10, 2007 4:07 PM, Gustavo Carneiro [EMAIL PROTECTED] wrote:
  On 10/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote:
 
Adam  I are now on #python-dev. Can you join?
  
   I think we successfully resolved this. Adam Olsen will produce a patch
   that allows one to specify a single file descriptor to which a zero
   byte will be written by the C-level signal handler. Twisted and PyGTK
   will have to coordinate about this file descriptor. Glyph believes
   this is possible and sufficient.
 
  Great that a solution was found at last.  But to be honest I don't
  understand why my patch (the second one) is not good.  Why is it
 preferable
  for libraries to provide their own file descriptor?

 So the app is in charge of creating the pipe if it wants it.

 We expect that if the pipe is always created, apps that assume only
 fds 0, 1 and 2 are open may get confused or accidentally close it,
 etc.


It's the first time I hear about this problem, but sounds plausible.

 Are people concerned about the overhead of always creating a pipe even it
  may not be used?  Always creating a pipe would at least avoid the need
 for
  any coordination between PyGTK and Twisted.

 I don't think that anyone involved though the need for coordination
 was an issue.


Yeah.   Thinking again on this, I don't think this is a problem, not because
coordination between PyGTK and Twisted can be done, but because no such
coordination is needed.  PyGTK can have one FD, Twisted another FD.  Since
only one of the main loops can be running, there's no conflict.

Please let this rest. We have a solution.


Sure.  :-)

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Gustavo Carneiro]

On 08/12/2007, Johan Dahlin [EMAIL PROTECTED] wrote:

 Guido van Rossum wrote:
  Adam, perhaps at some point (Monday?) we could get together on
  #python-dev and interact in real time on this issue. Probably even
  better on the phone. This offer is open to anyone who is serious about
  getting this resolved. Someone please take it -- I'm offering free
  consulting here!
 
  I'm curious -- is there anyone here who understands why [Py]GTK is
  using signals anyway? It's not like writing robust signal handling
  code in C is at all easy or obvious. If instead of a signal a file
  descriptor could be used, all problems would likely be gone.

 The timeout handler was added for KeyboardInterrupt to be able to work
 when
 you want to Ctrl-C yourself out of the gtk.main() loop.


Not only that, but pygtk is a generic module; who are we to forbid the usage
of signals if python itself allows it?  If we were to ignore signals then
sooner or later someone would come along and shout hey, signals work just
fine in pure python, so why did pygtk have to break my signals?.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Gustavo Carneiro]

On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote:

 On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote:
  Not only that, but pygtk is a generic module;

 What does generic mean in this context? Surely not that it applies
 to other libraries than GTK?


Well, actually this is also a PyGObject issue, not only PyGtk.  PyGObject
wraps GLib.  GLib was created to serve the needs of Gtk+, but is useful by
itself for writing portable programs.  Among other things, GLib offers a
generic main loop, which programs can use to do generic event-driven
programming, such as timeouts, polling file descriptors, and doing other
work when the main loop becomes idle.

You could argue that non-gtk programs are rare and we shouldn't worry too
much about them.  Maybe it's true, I don't know.

 who are we to forbid the usage
  of signals if python itself allows it?  If we were to ignore signals
 then
  sooner or later someone would come along and shout hey, signals work
 just
  fine in pure python, so why did pygtk have to break my signals?.

 Um, signals don't work just fine in pure Python. And I would argue
 they don't either in C. They are extremely subtle, and most code using
 signals is broken in some way. Just try to read the sigaction() man
 page and claim you understand it.

 Unfortunately, in Unix there are some conditions that can only be
 delivered using signals (e.g. SIGINTR, SIGTERM) and others for which
 your choices are either polling or signals (SIGCHILD, SIGWINCH).
 Traditionally, solutions based on select() or poll() with a short
 timeout (e.g. 20 or 100 ms) have worked well, as the number of
 instructions executed each time is really negligeable, and the
 response time is still reasonable on a human time scale. Unfortunately
 it seems recent developments in power management for ultra-low power
 devices have made this an issue again.

 Windows solves this more elegantly by having a unified wait for
 multiple conditions system call which can wait on I/O, semaphores,
 and other events (within the same process or coming from other
 processes). Unfortunately, in Unix, some events don't have a file
 descriptor associated with them, and for those select()/poll() cannot
 be used.

 The best solution I can think of is to add a new API that takes a
 signal and a file descriptor and registers a C-level handler for that
 signal which writes a byte to the file descriptor. You can then create
 a pipe, connect the signal handler to the write end, and add the read
 end to your list of file descriptors passed to select() or poll(). The
 handler must be written in C in order to avoid the race condition
 referred to by Glyph (signals arriving after the signal check in the
 VM main loop but before the select()/poll() system call is entered
 will not be noticed until the select()/poll() call completes).


Funny that everyone mentions this solution, as it is the solution
implemented by my patch :-)

Well, to be fair, it might not be _exactly_ what is implemented by the
patch.  Reading between the lines, I think what you mean is to have python's
C signal handler mostly untouched; it would only write a byte to a pipe _in
addition to_ the normal setting the flag and Py_AddPendingCall.

The patch I submitted, on the other hand, completely removes the vector of
flags and Py_AddPendingCall, and instead writes the number of the signal
that was raised into the pipe, and reads it back from the pipe in the Python
main loop.

Which is the best solution?  I think my patch fixes two problems: 1. the
need to have a FD to wake up poll() (t o fix the problem with what we are
discussing in this thread), and 2. make Python's signal handling more
reliable (not 100% reliable because it doesn't handle longer bursts of
signals than the pipe buffer can take, but at least is race free).

My solution is being reject because people are afraid to touch the signal
handling code, which has its faults, but well know faults.  But if I
refactor the patch to keep the crappy-but-sort-of-working signal code, and
only enhance it with the pipe, maybe it will more likely get accepted.

Do I understand correctly? :-)

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Gustavo Carneiro]

On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote:

 On Dec 8, 2007 9:20 AM, Guido van Rossum [EMAIL PROTECTED] wrote:
  The
  handler must be written in C in order to avoid the race condition
  referred to by Glyph (signals arriving after the signal check in the
  VM main loop but before the select()/poll() system call is entered
  will not be noticed until the select()/poll() call completes).

 Sorry, I misattributed this; Gustavo Carneiro pointed this out earlier
 in this thread.

 BTW, in the referenced post (also by Gustavo), I found this:

 
 According to [1], all python needs to do to avoid this problem is
 block all signals in all but the main thread; then we can guarantee
 signal handlers are always called from the main thread, and pygtk
 doesn't need a timeout.

 [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3
 

 Unfortunately I can't read [1] without having registered, so I can
 only guess what it says.


I can't read [1] either because the URL is not correct.  Sorry :P

But I know that Python already ensures that
 signals are only delivered to the main thread.


Are you sure?  In python code I see pthread_sigmask being checked for in
configure.in, but I can't find it being used anywhere (unless I'm grepping
for the wrong thing).

What you probably meant to say was python ensures that signals are only
processed from the main thread.

Except when python uses GNU PTH threads; there I do see signals being
delivered to the main thread.  But GNU pth are not real kernel threads,
anyway.

What am I missing?


I think Python could make sure (via pthread_sigmask) that signals are
blocked for all non-main threads created by Python.  Unfortunately it cannot
do the same for threads not created by Python.  I know at least one case
where threads are created behing Python's back: the GnomeVFS library.

Best regards,

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Gustavo Carneiro]

On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote:

 On Dec 8, 2007 9:57 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote:
  On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote:
 
   On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote:
Not only that, but pygtk is a generic module;
  
   What does generic mean in this context? Surely not that it applies
   to other libraries than GTK?
 
  Well, actually this is also a PyGObject issue, not only
 PyGtk.  PyGObject
  wraps GLib.  GLib was created to serve the needs of Gtk+, but is useful
 by
  itself for writing portable programs.  Among other things, GLib offers a
  generic main loop, which programs can use to do generic event-driven
  programming, such as timeouts, polling file descriptors, and doing other
  work when the main loop becomes idle.
 
  You could argue that non-gtk programs are rare and we shouldn't worry
 too
  much about them.  Maybe it's true, I don't know.
 
 
who are we to forbid the usage
of signals if python itself allows it?  If we were to ignore signals
  then
sooner or later someone would come along and shout hey, signals
 work
  just
fine in pure python, so why did pygtk have to break my signals?.
  
   Um, signals don't work just fine in pure Python. And I would argue
   they don't either in C. They are extremely subtle, and most code using
   signals is broken in some way. Just try to read the sigaction() man
   page and claim you understand it.
  
   Unfortunately, in Unix there are some conditions that can only be
   delivered using signals (e.g. SIGINTR, SIGTERM) and others for which
   your choices are either polling or signals (SIGCHILD, SIGWINCH).
   Traditionally, solutions based on select() or poll() with a short
   timeout (e.g. 20 or 100 ms) have worked well, as the number of
   instructions executed each time is really negligeable, and the
   response time is still reasonable on a human time scale. Unfortunately
   it seems recent developments in power management for ultra-low power
   devices have made this an issue again.
  
   Windows solves this more elegantly by having a unified wait for
   multiple conditions system call which can wait on I/O, semaphores,
   and other events (within the same process or coming from other
   processes). Unfortunately, in Unix, some events don't have a file
   descriptor associated with them, and for those select()/poll() cannot
   be used.
  
   The best solution I can think of is to add a new API that takes a
   signal and a file descriptor and registers a C-level handler for that
   signal which writes a byte to the file descriptor. You can then create
   a pipe, connect the signal handler to the write end, and add the read
   end to your list of file descriptors passed to select() or poll(). The
   handler must be written in C in order to avoid the race condition
   referred to by Glyph (signals arriving after the signal check in the
   VM main loop but before the select()/poll() system call is entered
   will not be noticed until the select()/poll() call completes).
 
  Funny that everyone mentions this solution, as it is the solution
  implemented by my patch :-)

 Mind pointing me to it again? I can't find it in the Python bug tracker.

  Well, to be fair, it might not be _exactly_ what is implemented by the
  patch.  Reading between the lines, I think what you mean is to have
 python's
  C signal handler mostly untouched; it would only write a byte to a pipe
 _in
  addition to_ the normal setting the flag and Py_AddPendingCall.

 Well, in many cases I see no problems with the current signal handler,
 and people are used to it, so I'm not sure what is gained by getting
 rid of it.

  The patch I submitted, on the other hand, completely removes the vector
 of
  flags and Py_AddPendingCall, and instead writes the number of the signal
  that was raised into the pipe, and reads it back from the pipe in the
 Python
  main loop.

 I believe Py_AddPendingCall was meant to be used by other places
 besides the signal handling.


True.  The patch does not remove Py_AddPendingCall, only stops using it for
signals.

 Which is the best solution?  I think my patch fixes two problems: 1. the
  need to have a FD to wake up poll() (t o fix the problem with what we
 are
  discussing in this thread), and 2. make Python's signal handling more
  reliable (not 100% reliable because it doesn't handle longer bursts of
  signals than the pipe buffer can take, but at least is race free).

 I think it's okay to drop signals if too many come. The FD should be
 put in non-blocking mode so the signal handler won't block forever.
 Does Unix even promise that a signal gets delivered twice if it gets
 sent quickly twice in a row?


Good point.

 My solution is being reject because people are afraid to touch the signal
  handling code, which has its faults, but well know faults.  But if I
  refactor the patch to keep the crappy-but-sort-of-working signal code,
 and
  only

Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Gustavo Carneiro]

On 09/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote:

 On Dec 8, 2007 3:57 PM, Adam Olsen [EMAIL PROTECTED] wrote:
 
  On Dec 8, 2007 4:28 PM, Guido van Rossum [EMAIL PROTECTED] wrote:
  
   On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote:
On Dec 8, 2007 2:56 PM,  [EMAIL PROTECTED] wrote:
 On 05:20 pm, [EMAIL PROTECTED] wrote:
 The best solution I can think of is to add a new API that takes a
 signal and a file descriptor and registers a C-level handler for
 that
 signal which writes a byte to the file descriptor. You can then
 create
 a pipe, connect the signal handler to the write end, and add the
 read
 end to your list of file descriptors passed to select() or
 poll(). The
 handler must be written in C in order to avoid the race condition
 referred to by Glyph (signals arriving after the signal check in
 the
 VM main loop but before the select()/poll() system call is
 entered
 will not be noticed until the select()/poll() call completes).

 This paragraph jogged my memory.  I remember this exact solution
 being
 discussed now, a year ago when I was last talking about these
 issues.

 There's another benefit to implementing a write-a-byte C signal
 handler.
 Without this feature, it wouldn't make sense to have passed the
 SA_RESTART flag to sigaction, because and GUIs written in Python
 could
 have spent an indefinite amount of time waiting to deliver their
 signal
 to Python code.  So, if you had to handle SIGCHLD in Python, for
 example, calls like file().write() would suddenly start raising a
 new
 exception (EINTR).  With it, you could avoid a whole class of
 subtle
 error-handling code in Twisted programs.
   
SA_RESTART still isn't useful.  The low-level poll call (not write!)
must stop and call back into python.  If that doesn't indicate an
error you can safely restart your poll call though, and follow it
 with
a (probably non-blocking) write.
  
   Can't say I understand all of this, but it does reiterate that there
   are more problems with signals than just the issue that Gustavo is
   trying to squash. The possibility of having *any* I/O interrupted is
   indeed a big worry. Though perhaps this could be alleviated by rigging
   things so that signals get delivered (at the C level) to the main
   thread and the rest of the code runs in a non-main thread?
 
  That's the approach my threading patch will take, although reversed
  (signals are handled by a background thread, leaving the main thread
  as the *main* thread.)

 Hm... Does this mean you're *always* creating an extra thread to handle
 signals?

  I share your concern about interrupting whatever random syscalls (not
  even limited to I/O!) that a library happens to use.
 
 
Note that the only reason to use C for a low-level handler here is
give access to sigatomic_t and avoid needing locks.  If you ran the
signal handler in a background thread (using sigwait to trigger
 them)
you could use a python handler.
  
   I haven't seen Gustavo's patch yet, but *my* reason for using a C
   handler was different -- it was because writing a byte to a pipe in
   Python would do nothing to fix Gustavo's issue.
  
   Looking at the man page for sigwait()  it could be an alternative
   solution, but I'm not sure how it would actually allow PyGTK to catch
   KeyboardInterrupt.
 
  My mail at [1] was referring to this.  Option 1 involved writing to a
  pipe that gets polled while option 2 requires we generate a new signal
  targeting the specific thread we want to interrupt.
 
  I'd like to propose an interim solution though: pygtk could install
  their own SIGINT handler during the gtk mainloop (or all gtk code?),
  have it write to a pipe monitored by gtk, and have gtk raise
  KeyboardInterrupt if it gets used.  This won't allow custom SIGINT
  handlers or any other signal handlers to run promptly, but it should
  be good enough for OLPC's use case.
 
 
  [1]
 http://mail.python.org/pipermail/python-dev/2007-December/075607.html

 Since OLPC has to use 2.5 they don't really have another choice
 besides this or making the timeout (perhaps much) larger -- I'm not
 going to risk a change as big as anything proposed here for 2.5.2, so
 nothing will change before 2.6.

 I've got to say that all the cross-referencing and asynchronous
 discussion here makes it *very* difficult to wrap my head around the
 various proposals. It also doesn't help that different participants
 appear to have different use cases in mind. E.g. do we care about
 threads started directly from C++ code? (These happen all the time at
 Google, but we don't care much about signals.) And what about
 restarting system calls (like Glyph brought up)?

 I've seen references to bug #1643738 which got a thumbs up from Tim
 Peters -- Adam, what do you think of that? I know it doesn't address
 Gustavo's issue but it seems useful in its own right.


That issue 

Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Gustavo Carneiro]

On 07/12/2007, Sean Reifschneider [EMAIL PROTECTED] wrote:

 On Thu, Dec 06, 2007 at 10:55:12PM -0700, Adam Olsen wrote:
 That's pretty much what issue1564547 does.  I think there's two marks

 Good point, I hadn't seen that before.

 * Using poll and fd's is pretty platform specific for what should be a
 general-purpose API

 I would say that this is an optimization that helps a specific set of
 platforms, including one that I think we really care about, the OLPC which
 needs it for decreased battery use.  It doesn't cause breakage of
 other platforms, it just may not help them.


Not only that, but current python signal handling is not theorethically
async safe; there are race conditions in the Py_AddPendingCalls API, and it
just happens to work most of the time.

BTW, the problem is described here:
http://mail.python.org/pipermail/python-dev/2006-September/068569.html

Think of even python select.poll and multiple threads.  If you have dozens
of threads, each blocked in some system call, when a signal arrives it will
interrupt one of the system calls, causing it to return EINTR, and then
python checks for signals.  Now imagine all the non-main threads are not
created by python; then one of the threads that wakes up could very well be
non-python one, and so python will never realize a signal was delivered.

The solution of blocking signals in all but the python thread(s) is not
feasible as we cannot control all threads that are created, some are created
by C libraries...

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] for loop with if filter

[Gustavo Carneiro]

On 16/11/2007, Benji York [EMAIL PROTECTED] wrote:

 Gustavo Carneiro wrote:
  I am finding myself often doing for loops over a subset of a list, like:
 
  for r in results:
  if r.numNodes != numNodes:
  continue
  # do something with r
 
  It would be nice if the plain for loop was as flexible as list
 comprehensions
  and allowed an optional if clause, like this:
 
  for r in results if r.numNodes == numNodes:
  # do something with r

 You can do the same today, sans sugar:

  for r in (s for s in results if s.numNodes == numNodes):
  # do something with r


Yes, I can do that, as well as I can use the 'continue' statement, but both
versions are slightly more verbose and less clear than what I propose.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] for loop with if filter

[Gustavo Carneiro]

I am finding myself often doing for loops over a subset of a list, like:

for r in results:
if r.numNodes != numNodes:
continue
# do something with r

It would be nice if the plain for loop was as flexible as list
comprehensions and allowed an optional if clause, like this:

for r in results if r.numNodes == numNodes:
# do something with r

Has this idea come up before?  Does anyone else like this idea?

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GC Changes

[Gustavo Carneiro]

On 02/10/2007, Adam Olsen [EMAIL PROTECTED] wrote:

 On 10/1/07, Greg Ewing [EMAIL PROTECTED] wrote:
  Justin Tulloss wrote:
   Would
   somebody care to give me a brief overview on how the current gc module
   interacts with the interpreter
 
  The cyclic GC kicks in when memory is running low. Since

 This isn't true at all.  It's triggered by heuristics based on the
 total number of allocated objects.  It doesn't know how much memory is
 available and is not called if an allocation fails.


Correct.  And that reminds me of the limitation of the the Python GC: it
doesn't take into account how much memory is being indirectly retained by a
Python Object.  Like in the example I already gave, gtk.gdk.Pixbuf can
easily hold hundreds of megabytes, yet the GC gives it as much consideration
as it does to a simple python integer object which is several orders of
magnitude smaller.

If someone wanted to improve the GC, that person should consider adding a
protocol for the GC to retrieve the ammount of memory indirectly held by a
python object, and take such memory into consideration in its heuristics.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GC Changes

[Gustavo Carneiro]

On 02/10/2007, Hrvoje Nikšić [EMAIL PROTECTED] wrote:

 On Tue, 2007-10-02 at 10:50 +0100, Gustavo Carneiro wrote:
  Correct.  And that reminds me of the limitation of the the Python GC:
  it doesn't take into account how much memory is being indirectly
  retained by a Python Object.  Like in the example I already gave,
  gtk.gdk.Pixbuf can easily hold hundreds of megabytes, yet the GC gives
  it as much consideration as it does to a simple python integer object
  which is several orders of magnitude smaller.

 That sounds like a case for the Pixbuf object to have a close method
 (not necessarily called that) that releases the resources.  The point of
 GC is that you normally don't care if memory is released sooner or
 later; for stuff you do care about, such as files, shared memory, or
 even large memory objects, there is always explicit management.
 cStringIO's close method provides a precedent.


I think close in real files is needed  not so much because you want to free
memory, but that you want to prevent data loss by flushing the file buffer
into the actual file at a certain point in time.  And I suspect that
cStringIO just added a close() method for compatibility with real files.
But of course I speculating here...

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GC Changes

[Gustavo Carneiro]

On 01/10/2007, Justin Tulloss [EMAIL PROTECTED] wrote:

 Hello,

 I've been doing some tests on removing the GIL, and it's becoming clear
 that some basic changes to the garbage collector may be needed in order for
 this to happen efficiently. Reference counting as it stands today is not
 very scalable.

 I've been looking into a few options, and I'm leaning towards the
 implementing IBMs recycler GC 
 (http://www.research.ibm.com/people/d/dfb/recycler-publications.html )
 since it is very similar to what is in place now from the users'
 perspective. However, I haven't been around the list long enough to really
 understand the feeling in the community on GC in the future of the
 interpreter. It seems that a full GC might have a lot of benefits in terms
 of performance and scalability, and I think that the current gc module is of
 the mark-and-sweep variety. Is the trend going to be to move away from
 reference counting and towards the mark-and-sweep implementation that
 currently exists, or is reference counting a firmly ingrained tradition?

 On a more immediately relevant note, I'm not certain I understand the full
 extent of the gc module. From what I've read, it sounds like it's fairly
 close to a fully functional GC, yet it seems to exist only as a
 cycle-detecting backup to the reference counting mechanism. Would somebody
 care to give me a brief overview on how the current gc module interacts with
 the interpreter, or point me to a place where that is done? Why isn't the
 mark-and-sweep mechanism used for all memory management?


The cyclic GC is just too slow to react and makes programmers mad.

For instance, in PyGtk we had a traditional problem with gtk.gdk.Pixbuf,
which is basically an object that wraps a raw RGB image.  When users deleted
such an object, which could sometimes comprise tens or hundreds of
megabytes, the memory was not relased until much much later.  That kind of
code ended up having to manually call gc.collect() to fix what was perceived
by most programmers as a memory leak, which kind of defeats the purpose of
a garbage collector.  This happened because PyGtk used to rely on the cyclic
GC doing its work.  Thankfully we moved away from that and now simple
reference counting can free a Pixbuf in most cases.

The cyclic GC is a very useful system, but it should only be used in
addition to, not instead of, reference counting.  At least that's my
personal opinion...

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] working with Python threads from C extension module?

[Gustavo Carneiro]

On 08/09/2007, Bill Janssen [EMAIL PROTECTED] wrote:

  Be convoluted yourself and do this:
 
  #define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) {
 Py_BEGIN_ALLOW_THREADS
  #define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } }
 
  (Untested, but I think it should work.)

 Yes, that had occurred to me.  We want the code inside the braces
 still to run if the locks aren't held, so something more like

   #define PySSL_BEGIN_ALLOW_THREADS { \
 PyThreadState *_save;  \
 if (_ssl_locks_count0) {_save =
 PyEval_SaveThread();}
   #define PySSL_BLOCK_THREADS   if
 (_ssl_locks_count0){PyEval_RestoreThread(_save)};
   #define PySSL_UNBLOCK_THREADS if (_ssl_locks_count0){_save =
 PyEval_SaveThread()};
   #define PySSL_END_ALLOW_THREADS   if
 (_ssl_locks_count0){PyEval_RestoreThread(_save);} \
  }

 would do the trick.  Unfortunately, this doesn't deal with the macro
 behaviour.  The user has turned on threading; they expect reads and
 writes to yield the GIL so that other threads can make progress.  But
 the fact that threading has been turned on after the SSL module has
 been initialized, means that threads don't work inside the SSL code.
 So the user's understanding of the system will be broken.

 No, I don't see any good way to fix this except to add a callback
 chain inside PyThread_init_thread, which is run down when threads are
 initialized.  Any module which needs to set up threads registers itself
 on that chain, and gets called as part of PyThread_init_thread.  But
 I'm far from the smartest person on this list :-), so perhaps someone
 else will see a good solution.


I think this is a helpful additional tool to solve threading problems.
Doesn't solve everything, but it certainly helps :-)

For instance, one thing it doesn't solve is when a library being wrapped can
be initialized with multithreading support, but only allows such
initialization as a very first API call; you can't initialize threading at
any arbitrary time during application runtime.  Unfortunately I don't think
there is any sane way to fix this problem :-(

This has got to be a problem with other extension modules linked to
 libraries which have their own threading abstractions.


Yes.

Another problem is that python extensions may not wish to incur performance
penalty of python threading calls.  For instance, pyorbit has these macros:

#define pyorbit_gil_state_ensure() (PyEval_ThreadsInitialized()?
(PyGILState_Ensure()) : 0)

#define pyorbit_gil_state_release(state) G_STMT_START { \
if (PyEval_ThreadsInitialized())\
PyGILState_Release(state);  \
} G_STMT_END

#define pyorbit_begin_allow_threads \
G_STMT_START {  \
PyThreadState *_save = NULL;\
if (PyEval_ThreadsInitialized())\
_save = PyEval_SaveThread();

#define pyorbit_end_allow_threads   \
if (PyEval_ThreadsInitialized())\
PyEval_RestoreThread(_save);\
} G_STMT_END

They all call PyEval_ThreadsInitialized() before doing anything thread
related to save some performance.  The other reason to do it this way is
that the Python API calls themselves abort if they are called with threading
not initialized.  It would be nice the upstream python GIL macros were more
like pyorbit and became no-ops when threading is not enabled.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 2.6 BaseException.message deprecation, really needed?

[Gustavo Carneiro]

In PyGObject we want to use a 'message' attribute in an exception defined by
us.  The name 'message' comes from a C API, therefore we would like to keep
it for easier mapping between C and Python APIs.  Why does Python have to
deprecate this attribute?

.../gobject/option.py:187: DeprecationWarning: BaseException.message has
been deprecated as of Python 2.6
 gerror.message = str(error)

--
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Instance variable access and descriptors

[Gustavo Carneiro]

 While you're at it, it would be nice to fix this ugly asymmetry I found in
descriptors.  It seems that descriptor's __get__ is called even when
accessed from a class rather than instance, but __set__ is only invoked from
instances, never from classes:

class Descr(object):
   def __get__(self, obj, objtype):
   print __get__ from instance %s, type %s % (obj, type)
   return foo

   def __set__(self, obj, value):
   print __set__ on instance %s, value %s % (obj, value)

class Foo(object):
   foo = Descr()

print Foo.foo # works

## doesn't work, goes directly to the class dict, not calling __set__
Foo.foo = 123

 Because of this problem, I may have to install properties into a class's
metaclass achieve the same effect that I expected to achieve with a simple
descriptor :-(


On 10/06/07, Aahz [EMAIL PROTECTED] wrote:


On Sun, Jun 10, 2007, Eyal Lotem wrote:

 Python, probably through the valid assumption that most attribute
 lookups go to the class, tries to look for the attribute in the class
 first, and in the instance, second.

 What Python currently does is quite peculiar!
 Here's a short description o PyObject_GenericGetAttr:

 A. Python looks for a descriptor in the _entire_ mro hierarchy
 (len(mro) class/type check and dict lookups).
 B. If Python found a descriptor and it has both get and set functions
 - it uses it to get the value and returns, skipping the next stage.
 C. If Python either did not find a descriptor, or found one that has
 no setter, it will try a lookup in the instance dict.
 D. If Python failed to find it in the instance, it will use the
 descriptor's getter, and if it has no getter it will use the
 descriptor itself.

Guido, Ping, and I tried working on this at the sprint for PyCon 2003.
We were unable to find any solution that did not affect critical-path
timing.  As other people have noted, the current semantics cannot be
changed.  I'll also echo other people and suggest that this discusion be
moved to python-ideas if you want to continue pushing for a change in
semantics.

I just did a Google for my notes from PyCon 2003 and it appears that I
never sent them out (probably because they aren't particularly
comprehensible).  Here they are for the record (from 3/25/2003):

'''
CACHE_ATTR is the name used to describe a speedup (for new-style classes
only) in attribute lookup by caching the location of attributes in the
MRO.  Some of the non-obvious bits of code:

* If a new-style class has any classic classes in its bases, we
can't do attribute caching (we need to weakrefs to the derived
classes).

* If searching the MRO for an attribute discovers a data descriptor (has
tp_descr_set), that overrides any attribute that might be in the instance;
however, the existence of tp_descr_get still permits the instance to
override its bases (but tp_descr_get is called if there is no instance
attribute).

* We need to invalidate the cache for the updated attribute in all derived
classes in the following cases:

* an attribute is added or deleted to the class or its base classes

* an attribute has its status changed to or from being a data
descriptor

This file uses Python pseudocode to describe changes necessary to
implement CACHE_ATTR at the C level.  Except for class Meta, these are
all exact descriptions of the work being done.  Except for class Meta the
changes go into object.c (Meta goes into typeobject.c).  The pseudocode
looks somewhat C-like to ease the transformation.
'''

NULL = object()

def getattr(inst, name):
isdata, where = lookup(inst.__class__, name)
if isdata:
descr = where[name]
if hasattr(descr, __get__):
return descr.__get__(inst)
else:
return descr
value = inst.__dict__.get(name, NULL)
if value != NULL:
return value
if where == NULL:
raise AttributError
descr = where[name]
if hasattr(descr, __get__):
value = descr.__get__(inst)
else:
value = descr
return value

def setattr(inst, name, value):
isdata, where = lookup(inst.__class__, name)
if isdata:
descr = where[name]
descr.__set__(inst, value)
return
inst.__dict__[name] = value

def lookup(cls, name):
if cls.__cache__ != NULL:
pair = cls.__cache__.get(name)
else:
pair = NULL
if pair:
return pair
else:
for c in cls.__mro__:
where = c.__dict__
if name in where:
descr = where[name]
isdata = hasattr(descr, __set__)
pair = isdata, where
break
else:
pair = False, NULL
if cls.__cache__ != NULL:
cls.__cache__[name] = pair
return pair


'''
These changes go into typeobject.c; they are not a complete
description of what happens during creation/updates, only the
changes necessary to implement CACHE_ATTRO.
'''

from types import ClassType

class Meta(type):
def _invalidate(cls, 

Re: [Python-Dev] Fwd: Instance variable access and descriptors

[Gustavo Carneiro]

 I have to agree with you.  If removing support for
self.__dict__['propertyname'] (where propertyname is also the name of a
descriptor) is the price to pay for significant speedup, so be it.  People
doing that are asking for trouble anyway!

On 10/06/07, Eyal Lotem [EMAIL PROTECTED] wrote:


On 6/10/07, Phillip J. Eby [EMAIL PROTECTED] wrote:
 At 12:23 AM 6/10/2007 +0300, Eyal Lotem wrote:
 A. It will break code that uses instance.__dict__['var'] directly,
 when 'var' exists as a property with a __set__ in the class. I believe
 this is not significant.
 B. It will simplify getattr's semantics. Python should _always_ give
 precedence to instance attributes over class ones, rather than have
 very weird special-cases (such as a property with a __set__).

 Actually, these are features that are both used and desirable; I've
 been using them both since Python 2.2 (i.e., for many years
 now).  I'm -1 on removing these features from any version of Python,
even 3.0.

It is the same feature, actually, two sides of the same coin.
Why do you use self.__dict__['propertyname'] when you can use
self._propertyname?
Why even call the first form, which is both longer and causes
performance problems a feature?


 C. It will greatly speed up instance variable access, especially when
 the class has a large mro.

 ...at the cost of slowing down access to properties and __slots__, by
 adding an *extra* dictionary lookup there.
It will slow down access to properties - but that slowdown is
insignificant:
A. The vast majority of lookups are *NOT* of properties. They are the
rare case and should not be the focus of optimization.
B. Property access involves calling Python functions - which is
heavier than a single dict lookup.
C. The dict lookup to find the property in the __mro__ can involve
many dicts (so in those cases adding a single dict lookup is not
heavy).

 Note, by the way, that if you want to change attribute lookup
 semantics, you can always override __getattribute__ and make it work
 whatever way you like, without forcing everybody else to change *their*
code.
If I write my own __getattribute__ I lose the performance benefit that
I am after.
I do agree that code shouldn't be broken, that's why a transitional
that requires using __fastlookup__ can be used (Unfortunately, from
__future__ cannot be used as it is not local to a module, but to a
class hierarchy - unless one imports a feature from __future__ into a
class).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com





--
Gustavo J. A. M. Carneiro
INESC Porto
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Super PEP

[Gustavo Carneiro]

On 29/04/07, James Y Knight [EMAIL PROTECTED] wrote:



On Apr 28, 2007, at 10:43 PM, Calvin Spealman wrote:
 Abstract
 

 The PEP defines the proposal to enhance the super builtin to work
 implicitly
 upon the class within which it is used and upon the instance the
 current
 function was called on. The premise of the new super usage
 suggested is as
 follows:

 super.foo(1, 2)

 to replace the old:

 super(Foo, self).foo(1, 2)


 Rationale
 =

 The current usage of super requires an explicit passing of both the
 class and
 instance it must operate from, requiring a breaking of the DRY
 (Don't Repeat
 Yourself) rule. This hinders any change in class name, and is often
 considered
 a wart by many.

This is only a halfway fix to DRY, and it really only fixes the less
important half. The important problem with super is that it
encourages people to write incorrect code by requiring that you
explicitly specify an argument list.



Since calling super with any

arguments other than the exact same arguments you have received is
nearly always wrong,



 Erm.  Excuse me, but are you saying this code is wrong?

class Rectangle:
   def __init__(self, width, height):
   self.width = width
   self.height = height

class Square:
   def __init__(self, side):
   Rectangle.__init__(self, side, side)

Or are you even saying this type of code is rare?  I would disagree with
both statements, therefore I also disagree with your recommendation.

--
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3000 PEP: Postfix type declarations

[Gustavo Carneiro]

On 4/1/07, Georg Brandl [EMAIL PROTECTED] wrote:
[...]


Example
===

This is the standard ``os.path.normpath`` function, converted to type
declaration
syntax::

 def normpathƛ(path✎)✎:
 Normalize path, eliminating double slashes, etc.
 if path✎ == '':
 return '.'
 initial_slashes✓ = path✎.startswithƛ('/')✓
 # POSIX allows one or two initial slashes, but treats three or
more
 # as single slash.
 if (initial_slashes✓ and
 path✎.startswithƛ('//')✓ and not path✎.startswithƛ('///')✓)✓:
 initial_slashesℕ = 2
 comps♨ = path✎.splitƛ('/')♨
 new_comps♨ = []♨
 for comp✎ in comps♨:
 if comp✎ in ('', '.')⒯:
 continue
 if (comp✎ != '..' or (not initial_slashesℕ and not
new_comps♨)✓ or
  (new_comps♨ and new_comps♨[-1]✎ == '..')✓)✓:
 new_comps♨.appendƛ(comp✎)
 elif new_comps♨:
 new_comps♨.popƛ()✎
 comps♨ = new_comps♨
 path✎ = '/'.join(comps♨)✎
 if initial_slashesℕ:
 path✎ = '/'*initial_slashesℕ + path✎
 return path✎ or '.'

As you can clearly see, the type declarations add expressiveness, while at
the
same time they make the code look much more professional.



 Is this supposed to be a joke?  Please tell me this isn't a real PEP.
While I'm all for allowing unicode identifiers in Python, postfix type
annotations make Python look like Perl.  And how can you claim this code is
more readable?  It certainly is _less_ readable, not more.

 I agree that Python should support type annotations, but they should be
optional and only present at the function interfaces, i.e. specify the type
in the function parameter lists, like in plain old C.

 +1 from me for allowing unicode identifiers.

 -MAXVOTE for type annotations in identifiers.

--
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3000 PEP: Postfix type declarations

[Gustavo Carneiro]

On 4/1/07, Gustavo Carneiro [EMAIL PROTECTED] wrote:


On 4/1/07, Georg Brandl [EMAIL PROTECTED] wrote:
[...]

 Example
 ===

 This is the standard ``os.path.normpath`` function, converted to type
 declaration
 syntax::

  def normpathƛ(path✎)✎:
  Normalize path, eliminating double slashes, etc.
  if path✎ == '':
  return '.'
  initial_slashes✓ = path✎.startswithƛ('/')✓
  # POSIX allows one or two initial slashes, but treats three or
 more
  # as single slash.
  if (initial_slashes✓ and
  path✎.startswithƛ('//')✓ and not
 path✎.startswithƛ('///')✓)✓:
  initial_slashesℕ = 2
  comps♨ = path✎.splitƛ('/')♨
  new_comps♨ = []♨
  for comp✎ in comps♨:
  if comp✎ in ('', '.')⒯:
  continue
  if (comp✎ != '..' or (not initial_slashesℕ and not
 new_comps♨)✓ or
   (new_comps♨ and new_comps♨[-1]✎ == '..')✓)✓:
  new_comps♨.appendƛ(comp✎)
  elif new_comps♨:
  new_comps♨.popƛ()✎
  comps♨ = new_comps♨
  path✎ = '/'.join(comps♨)✎
  if initial_slashesℕ:
  path✎ = '/'*initial_slashesℕ + path✎
  return path✎ or '.'

 As you can clearly see, the type declarations add expressiveness, while
 at the
 same time they make the code look much more professional.


  Is this supposed to be a joke?



 /me ashamed for not having noticed the date of this PEP... :P

--
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rationale for NamedTemporaryFile?

[Gustavo Carneiro]

On 3/21/07, Scott Dial [EMAIL PROTECTED] wrote:


Greg Ewing wrote:
 A tangential question -- why are TemporaryFile and
 NamedTemporaryFile named in TitleCase, when they're
 functions and not classes?


I wondered the same. At first draft of my email I wrote class
operating under the assumption that only classes got to be camel-cased.
If I had to guess, the rationale was that they are simply wrappers a
class. Nevertheless, tempfile dates well back before the PEP 8 style
guidelines. I think consistency would dictate that a newly added
function should match the other ones, but I have no strong feelings
about this.



 I wouldn't bet on that.  The ElementTree XML module was added to Python
2.5 and it does not follow PEP 8 style (method names are not words separated
by underscores).

--
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pydoc II

[Gustavo Carneiro]

On 3/11/07, Laurent Gautier [EMAIL PROTECTED] wrote:[...]


A prototype is being worked on, and I have been looking at code and/or
functionalities in pydoc, epydoc, pydoctor, and ipython (for the
interactive console), and there a lot of nice things in all these
efforts. The name of the game is now to have something that likely to
offer the best of all (although it can be an easy way to get no one
happy in the end).

Hopefully someone kept reading up to here...

Some of the key points so far:

library:
- ability to browse documentation inside module/package files without
loading them



 Most importantly, we also need the reserve: ability to attach external
documentation to modules/packages without increasing their size.

 For instance, one of the reasons no one ever bothered to add pydoc
documentation to PyGTK (beyond a few docstrings that can be generated in
runtime) is that the entire documentation for PyGTK would be far too large
to place directly into the module readonly data segment.

 Documentation should be able to live in another file, separate from the
module source code, and loaded on demand.  Loading the pydoc file should be
preferably done through read-only memory mapping a file.

--
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New syntax for 'dynamic' attribute access

[Gustavo Carneiro]

On 2/11/07, Ben North [EMAIL PROTECTED] wrote:


Hi,

A few days ago I posted to python-ideas with a suggestion for some new
Python syntax, which would allow easier access to object attributes
where the attribute name is known only at run-time.  For example:

   setattr(self, method_name, getattr(self.metadata, method_name))

from Lib/distutils/dist.py could be rewritten

   self.(method_name) = self.metadata.(method_name)

The new syntax would also be usable in augmented assignments, as in

   obj.(attr_name) += 1



 -1 from me.  It does not solve a common problem, therefore it does not
deserve a special syntax.   Moreover, the existing syntax may be longer to
type but is far more readable.

--
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem with signals in a single threaded application

[Gustavo Carneiro]

On 1/24/07, Martin v. Löwis [EMAIL PROTECTED] wrote:


Gustavo Carneiro schrieb:
What about http://www.python.org/sf/1564547 ?  It tries to solve a
 different problem, but I think it also fixes this one; at least as much
 as possible with the braindead unix signal programming interface...

I'm sceptical. It is way too much code for me to review, so I'm unable
to comment whether it fixes the problem under discussion.



 Due to a bug, my patch  didn't fix this bug.  But I fixed the patch and it
now fixes both my problem and Ulisses'.

I feel that

this problem should find a much simpler solution.



  The problem is that if you apply Ulisses' patch then my patch, Ulisses'
changes will simply disappear because my patch handles signals and a much
safer way, completely bypassing the 'add/make pending calls' system, since
this system is patently *not* async safe, no matter how much you tweak it.

 Yes, I know my patch is not very small (though not that big either), but
the signal module was in dire need of refactoring.  And you can observe how
much simpler the signal module becomes after that patch.

 Regards.

--
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem with signals in a single threaded application

[Gustavo Carneiro]

On 1/24/07, Martin v. Löwis [EMAIL PROTECTED] wrote:


Ulisses Furquim schrieb:
 I've read some threads about signals in the archives and I was under
 the impression signals should work reliably on single-threaded
 applications. Am I right?  I've thought about a way to fix this, but I
 don't know what is the current plan for signals support in python, so
 what can be done?

I agree it's a bug, and I agree with your proposed analysis. Please
try to come up with a patch (e.g. by putting a while(is_tripped) loop
around the for loop). Also make sure you include test case.



  What about http://www.python.org/sf/1564547 ?  It tries to solve a
different problem, but I think it also fixes this one; at least as much as
possible with the braindead unix signal programming interface...

--
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Polling with Pending Calls?

[Gustavo Carneiro]

 This patch may interest you: http://www.python.org/sf/1564547

 Not sure it completely solves your case, but it's at least close to your
problem.

On 12/4/06, Tony Nelson [EMAIL PROTECTED] wrote:


I think I have a need to handle *nix signals through polling in a library.
It looks like chaining Pending Calls is almost the way to do it, but I see
that doing so would make the interpreter edgy.

The RPM library takes (steals) the signal handling away from its client
application.  It has good reason to be so paranoid, but it breaks the
handling keyboard interrupts, especially if rpmlib is used in the normal
way:  opened at the beginning, various things are done by the app, closed
at the end.  If there is an extended period in the middle where no calls
are made to rpmlib (say, in yum during the downloading of packages or
package headers), then responst to a keyboard interrupt can be delayed for
/minutes/!  Yum is presently doing something awful to work around that
issue.

It is possible to poll rpmlib to find if there is a pending keyboard
interrupt.  Client applications could have such polls sprinkled throughout
them.  I think getting yum, for example, to do that might be politically
difficult.  I'm hoping to propose a patch to rpmlib's Python bindings to
do
the polling automagically.

Looking at Python's normal signal handling, I see that Py_AddPendingCall()
and Py_MakePendingCalls(), and  PyEvel_EvalFrameEx()'s ticker check are
how
signals and other async events are done.  I could imagine making rpmlib's
Python bindings add a Pending Call when the library is loaded (or some
such), and that Pending Call would make a quick check of rpmlib's caught
signals flags and then call Py_AddPendingCall() on itself.  It appears
that
this would work, and is almost the expected thing to do, but unfortunately
it would cause the ticker check to think that Py_MakePendingCalls() had
failed and needed to be called again ASAP, which would drastically slow
down the interpreter.

Is there a right way to get the Python interpreter to poll something, or
should I look for another approach?

[I hope this message doesn't spend too many days in the grey list limbo.]
--

TonyN.:'The Great Writ mailto:[EMAIL PROTECTED]
  '  is no more. http://www.georgeanelson.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com





--
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

- http://www.python.org/sf/1564547

-- 
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Grammar change in classdef

[Gustavo Carneiro]

On 9/16/06, Talin [EMAIL PROTECTED] wrote:
 Nick Coghlan wrote:
  As for the reason: it makes it possible to use the same style for classes
  without bases as is used for functions without arguments. Prior to this
  change, there was a sharp break in the class syntax, such that if you got 
  rid
  of the last base class you had to get rid of the parentheses as well.

 Is the result a new-style or classic-style class? It would be nice if
 using the empty parens forced a new-style class...

  That was my first thought as well.  Unfortunately a quick test shows
that class Foo(): creates an old style class instead :(

-- 
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/12/06, Adam Olsen [EMAIL PROTECTED] wrote:
 On 9/12/06, Gustavo Carneiro [EMAIL PROTECTED] wrote:
  On 9/12/06, Adam Olsen [EMAIL PROTECTED] wrote:
   My previous mention of using a *single* flag may survive corruption
   simply because we can tolerate false positives.  Signal handlers would
   write 0x, the poll loop would check if *any* bit is set.  If
   so, write 0x0, read off the fd, then loop around and check it again.
   If the start of the read() acts as a write-barrier it SHOULD guarantee
   we don't miss any positive writes.
 
Why write 0x?  Why can't the variable be of a volatile
  char type?  Assuming sizeof(char) == 1, please don't tell me
  architecture XPTO will write the value 4 bits at a time! :P

 Nope.  It'll write 32 bits, then break that up into 8 bits :)
 Although, at the moment I can't fathom what harm that would cause...

  Hmm... it means that to write those 8 bits the processor / compiler
may need to 1. read 32 bits from memory to a register, 2. modify 8
bits of the register, 3. write back those 32 bits.  Shouldn't affect
our case, but maybe it's better to use sig_atomic_t in any case.

 For the record, all volatile does is prevent compiler reordering
 across sequence points.

  It makes the compiler aware the value may change any time, outside
the current context/function, so it doesn't assume a constant value
and always re-reads it from memory instead of assuming a value from a
register is correct.

  static volatile char signal_flag;
  static int signal_pipe_r, signal_pipe_w;
 
  PyErr_CheckSignals()
  {
if (signal_flag) {
   char signum;
   signal_flag = 0;
   while (read(signal_pipe_r, signum, 1) == 1)
   process_signal(signum);
}
  }

 I'd prefer this to be a while (signal_flag) instead, although it
 should technically work either way.

  I guess we can use while instead of if.


  static void
  signal_handler(int signum)
  {
 char signum_c = signum;
 signal_flag = 1;
 write(signal_pipe_w, signum_c, 1);
  }

 This is wrong.  PyErr_CheckSignals could check and clear signal_flag
 before you reach the write() call.  signal_flag = 1 should come
 after.

  Yes, good catch.

  I don't understand the memory barrier concern in your other email.
I know little on the subject, but from what I could find out memory
barriers are used to avoid reordering of multiple read and write
operations.  However, in this case we have a single value at stake,
there's nothing to reorder.

  Except perhaps that signal_flag = 0 could be delayed... If it is
delayed until after the while (read (...)...) loop below we could get
in trouble. I see your point now... :|

  But I think that a system call has to act as memory barrier,
forcefully, because the CPU has to jump into kernelspace, a completely
different context, it _has_ to flush pending memory operations sooner
or later.

Round two:

static volatile sig_atomic_t signal_flag;
static int signal_pipe_r, signal_pipe_w;

PyErr_CheckSignals()
{
   while (signal_flag) {
 char signum;
 signal_flag = 0;
 while (read(signal_pipe_r, signum, 1) == 1)
 process_signal(signum);
   }
}

static void
signal_handler(int signum)
{
char signum_c = signum;
write(signal_pipe_w, signum_c, 1);
signal_flag = 1;
}

-- 
Gustavo J. A. M. Carneiro
The universe is always one step beyond logic.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/12/06, Adam Olsen [EMAIL PROTECTED] wrote:
 On 9/12/06, Greg Ewing [EMAIL PROTECTED] wrote:
  Adam Olsen wrote:
 
   That brings you back to how you access the flags variable.
 
  The existing signal handler sets a flag, doesn't it?
  So it couldn't be any more broken than the current
  implementation.
 
  If we get too paranoid about this, we'll just end
  up deciding that signals can't be used for anything,
  at all, ever. That doesn't seem very helpful,
  although techically I suppose it would solve
  the problem. :-)
 
  My own conclusion from all this is that if you
  can't rely on writing to a variable in one part
  of your program and reading it back in another,
  then computer architectures have become far
  too clever for their own good. :-(

 They've been that way for a long, long time.  The irony is that x86 is
 immensely stupid in this regard, and as a result most programmers
 remain unaware of it.

 Other architectures have much more interesting read/write and cache
 reordering semantics, and the code is certainly broken there.  C
 leaves it undefined with good reason.

 My previous mention of using a *single* flag may survive corruption
 simply because we can tolerate false positives.  Signal handlers would
 write 0x, the poll loop would check if *any* bit is set.  If
 so, write 0x0, read off the fd, then loop around and check it again.
 If the start of the read() acts as a write-barrier it SHOULD guarantee
 we don't miss any positive writes.

  Why write 0x?  Why can't the variable be of a volatile
char type?  Assuming sizeof(char) == 1, please don't tell me
architecture XPTO will write the value 4 bits at a time! :P

  I see your point of using a flag to avoid the read() syscall most of
the time.  Slightly more complex, but possibly worth it.

  I was going to describe a possible race condition, then wrote the
code below to help explain it, modified it slightly, and now I think
the race is gone.  In any case, the code might be helpful to check if
we are in sync.  Let me know if you spot any  race condition I missed.


static volatile char signal_flag;
static int signal_pipe_r, signal_pipe_w;

PyErr_CheckSignals()
{
  if (signal_flag) {
 char signum;
 signal_flag = 0;
 while (read(signal_pipe_r, signum, 1) == 1)
 process_signal(signum);
  }
}

static void
signal_handler(int signum)
{
   char signum_c = signum;
   signal_flag = 1;
   write(signal_pipe_w, signum_c, 1);
}
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/11/06, Adam Olsen [EMAIL PROTECTED] wrote:
 Now on to my new proposal.  I do still use write().  If you can't
 accept that I think we should rip signals out entirely, just let them
 kill the process.  Not a reliable feature of any OS.

 We create a single pipe and use it for all signals.  We never release
 it, instead letting the OS do it when the process gets cleaned up.  We
 write the signal number to it as a byte (assuming there's at most 256
 unique signals).

 This much would allow a GUI's poll loop to wake up when there is a
 signal, and give control back to the python main loop, which could
 then read off the signals and queue up their handler functions.

  I like this approach.  Not only we would get a poll-able file
descriptor to notify a GUI main loop when signals arrive, we'd also
avoid the lack of async safety in Py_AddPendingCall /
Py_MakePendingCalls which affects _current_ Python code.

  Note that the file descriptor of the read end of the pipe has to
become a public Python API so that 3rd party extensions may poll it.
This is crucial.


 The only problem is when there is no GUI poll loop.  We don't want
 python to have to poll the fd, we'd rather it just check a variable.
 Is it possible to set/clear a flag in a sufficiently portable
 (reentrant-safe, non-blocking, thread-safe) fashion?

  It's simple.  That pipe file descriptor has to be changed to
non-blocking mode in both ends of the pipe, obviously, with fcntl.
Then, to find out whether a signal happened or not we modify
PyErr_CheckSignals() to try to read from the pipe.  If it reads bytes
from the pipe, we process the corresponding python signal handlers or
raise KeyboardInterrupt.  If the read() syscall returns zero bytes
read, we know no signal was delivered and move on.

  The only potential problem left is that, by changing the pipe file
descriptor to non-blocking mode we can only write as many bytes to it
without reading from the other side as the pipe buffer allows.  If a
large number of signals arrive very quickly, that buffer may fill and
we lose signals.  But I think the default buffer should be more than
enough.  And normally programs don't receive lots of signals in a
small time window.  If it happens we may lose signals, but that's very
rare, and who cares anyway.

  Regards.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/9/06, Jan Kanis [EMAIL PROTECTED] wrote:
 At the risk of waking up a thread that was already declared dead, but
 perhaps this is usefull.

 So, what happens is pythons signal handler sets a flag and registrers a
 callback. Then the main thread should check the flag and make the callback
 to actually do something with the signal. However the main thread is
 blocked in GTK and can't check the flag.

 Nick Maclaren wrote:
 ...lots of reasons why you can't do anything reliably from within a signal
 handler...

 As far as I understand it, what could work is this:
 -PyGTK registrers a callback.
 -Pythons signal handler does not change at all.
 -All threads that run in the Python interpreter occasionally check the
 flag which the signal handler sets, like the main thread does nowadays. If
 it is set, the thread calls PyGTKs callback. It does not do anything else
 with the signal.
 -PyGTKs callback wakes up the main thread, which actually handles the
 signal just like it does now.

 PyGTKs callback could be called from any thread, but it would be called in
 a normal context, not in a signal handler. As the signal handler does not
 change, the risk of breaking anything or causing chaos is as large/small
 as it is under the current scheme.

 However, PyGTKs problem does get
 solved, as long as there is _a_ thread that returns to the interpreter
 within some timeframe. It seems plausible that this will happen.

  No, it is not plausible at all.  For instance, the GnomeVFS library
usually has a pool of thread, not doing anything, waiting for some VFS
task.  It is likely that a signal will be delivered to one of these
threads, which know nothing about Python, and sit idle most of the
time.

  Regards.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/9/06, Adam Olsen [EMAIL PROTECTED] wrote:
 On 9/8/06, Adam Olsen [EMAIL PROTECTED] wrote:
  Ensuring modifications to that array are atomic would be tricky, but I
  think it would be doable if we use a read-copy-update approach (with
  two alternating signal handler functions).  Not sure how to ensure
  there's no currently running signal handlers in another thread though.
   Maybe have to rip the atomic read/write stuff out of the Linux
  sources to ensure it's *always* defined behavior.

 Doh, except that's exactly what sig_atomic_t is for.  Ah well, can't
 win them all.

From the glibc manual:

To avoid uncertainty about interrupting access to a variable, you can
use a particular data type for which access is always atomic:
sig_atomic_t. Reading and writing this data type is guaranteed to
happen in a single instruction, so there's no way for a handler to run
in the middle of an access.


  So, no, this is certainly not the same as linux kernel atomic
operations, which allow you to do more interesting stuff like,
test-and-clear, or decrement-and-test atomically.  glib has those too,
and so does mozilla's NSPR, but only on a few architectures does it do
it without using mutexes.  for instance, i686 onwards don't require
mutexes, only special instructions, but i386 requires mutexes.  And we
all know mutexes in signal handlers cause deadlocks :-(

  And, yes, Py_AddPendingCall and Py_MakePendingCalls are most
certainly not async safe!  Just look at the source code of
Py_MakePendingCalls and you'll see an interesting comment...
Therefore, discussions about signal safety in whatever new API we may
add to Python should be taken with a grain of salt.

  Regards.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/9/06, Nick Maclaren [EMAIL PROTECTED] wrote:
 I was hoping to have stopped, but here are a few comments.

 I agree with Jan Kanis.  That is the way to tackle this one.

  Alas, it doesn't work in practice, as I already replied.

[...]
 Despite the implication, the code of Py_AddPendingCall is NOT safe
 against simultaneous registration.  It is just plain broken, I am
 afraid.  The note starting Darn should be a LOT stronger :-)

  Considering that this code has existed for a very long time, and
that it isn't really safe, should we even bother to try to make
signals 100% reliable?

  I remember about a security-related module (bastion?) that first
claimed to allow execution of malicious code while protecting the
system; later, they figured out it wasn't really safe, and couldn't be
safe, so the documentation was simply changed to state not to use that
module if you need real security.

  I see the same problem here.  Python signal handling isn't _really_
100% reliable.  And it would be very hard to make Py_AddPendingCall /
Py_MakePendingCalls completely reliable.

But let's think for a moment.  Do we really _need_ to make Python unix
signal handling 100% reliable?  What are the uses for signals?  I can
only understand a couple of uses: handling of SIGINT for generating
KeyboardInterrupt [1], and handling of fatal errors like SIGSEGV in
order to show a crash dialog and bug reporting tool.  The second use
case doesn't demand 100% reliability.  The second use case is
currently being handled also in recent Ubuntu Linux through
/proc/sys/kernel/crashdump-helper.  Other notable uses that I see of
signals are sending SIGUSR1 or SIGHUP to a daemon to make it reload
its configuration.  But any competent programmer already knows how to
make the program use local sockets instead.


[1] Although ideally Python wouldn't even have KeyboardInterrupt and
just die on Ctrl-C like any normal program.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/5/06, Adam Olsen [EMAIL PROTECTED] wrote:
 On 9/4/06, Gustavo Carneiro [EMAIL PROTECTED] wrote:
Now, we've had this API for a long time already (at least 2.5
  years).  I'm pretty sure it works well enough on most *nix systems.
  Event if it works 99% of the times, it's way better than *failing*
  *100%* of the times, which is what happens now with Python.

 Failing 99% of the time is as bad as failing 100% of the time, if your
 goal is to eliminate the short timeout on poll().  1% is quite a lot,
 and it would probably have an annoying tendency to trigger repeatedly
 when the user does certain things (not reproducible by you of course).

 That said, I do hope we can get 100%, or at least enough nines that we
 can increase the timeout significantly.

  Anyway, I was speaking hypothetically.  I'm pretty sure writing to a
pipe is async signal safe.  It is the oldest trick in the book,
everyone uses it.  I don't have to see a written signed contract to
know that it works.

  Here's a list of web sites google found me that talk about this problem:

This one describes the pipe writing technique:
  http://www.cocoadev.com/index.pl?SignalSafety

This one presents a list of The only routines that POSIX guarantees
to be Async-Signal-Safe:
  
http://docsun.cites.uiuc.edu/sun_docs/C/solaris_9/SUNWdev/MTP/p40.html#GEN-95948

Also here:

http://www.cs.usyd.edu.au/cgi-bin/man.cgi?section=5topic=attributes

  This is all the evidence that I need.  And again I reiterate that
whether or not async safety can be achieved in practice for all
platforms is not Python's problem.  Although I believe writing to a
pipe is 100% reliable for most platforms.  Even if it is not, any
mission critical application relying on signals for correct behaviour
should be rewritten to use unix sockets instead; end of argument.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/4/06, Nick Maclaren [EMAIL PROTECTED] wrote:
 Gustavo Carneiro [EMAIL PROTECTED] wrote:
I am now thinking of something along these lines:
  typedef void (*PyPendingCallNotify)(void *user_data);
  PyAPI_FUNC(void) Py_AddPendingCallNotify(PyPendingCallNotify callback,
  void *user_data);
  PyAPI_FUNC(void) Py_RemovePendingCallNotify(PyPendingCallNotify
  callback, void *user_data);

 Why would that help?  The problems are semantic, not syntactic.

 Anthony Baxter isn't exaggerating the problem, despite what you may
 think from his posting.

  You guys are tough customers to please.  I am just trying to solve a
problem here, not create a new one; you have to believe me.

  OK, let's review what we know about current python, signals, and threads:

 1. Python launches threads without touching sigprocmask;
 2. Python installs signal handlers for all signals;
 3. Signals can be delivered to any thread, let's assume (because
of point #1 and not others not mentioned) that we have no control over
which threads receive which signals, might as well be random for all
we know;
 4. Python signal handlers do almost nothing: just sets a flag,
and calls Py_AddPendingCall, to postpone the job of handling a signal
until a safer time.
 5. The function Py_MakePendingCalls() should eventually get
called at a safer time by user or python code.
 6. It follows that until Py_MakePendingCalls() is called, the
signal will not be handled at all!

  Now, back to explaining the problem.

 1. In PyGTK we have a gobject.MainLoop.run() method, which blocks
essentially forever in a poll() system call, and only wakes if/when it
has to process timeout or IO event;
 2. When we only have one thread, we can guarantee that e.g.
SIGINT will always be caught by the thread running the
g_main_loop_run(), so we know poll() will be interrupted and a EINTR
will be generated, giving us control temporarily back to check for
python signals;
 3. When we have multiple thread, we cannot make this assumption,
so instead we install a timeout to periodically check for signals.

  We want to get rid of timeouts.  Now my idea: add a Python API to say:
 dear Python, please call me when you start having pending calls,
even if from a signal handler context, ok?

From that point on, signals will get handled by Python, python calls
PyGTK, PyGTK calls a special API to safely wake up the main loop even
from a thread or signal handler, then main loop checks for signal by
calling PyErr_CheckSignals(), it is handled by Python, and the process
lives happily ever after, or die trying.

  I sincerely hope my explanation was satisfactory this time.

  Best regards.


PS: there's a funny comment in Py_AddPendingCall that suggests it is
not very safe against reentrancy problems:

/* XXX Begin critical section */
/* XXX If you want this to be safe against nested
   XXX asynchronous calls, you'll have to work harder! */

Are signal handlers guaranteed to not be interrupted by another
signal, at least?  What about threads?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

  In GLib we have a child watch notification feature that relies on
the following signal handler:

static void
g_child_watch_signal_handler (int signum)
{
  child_watch_count ++;

  if (child_watch_init_state == CHILD_WATCH_INITIALIZED_THREADED)
{
  write (child_watch_wake_up_pipe[1], B, 1);
}
  else
{
  /* We count on the signal interrupting the poll in the same thread.
   */
}
}

  Now, we've had this API for a long time already (at least 2.5
years).  I'm pretty sure it works well enough on most *nix systems.
Event if it works 99% of the times, it's way better than *failing*
*100%* of the times, which is what happens now with Python.

  All I ask is an API to add a callback that Python signal handlers
call, from signal context.  That much I'm sure is safe.  What happens
from there on will be out of Python's hands, so Python
purist^H^H^H^H^H^H developers cannot be blamed for anything that
happens next.  You can laugh at PyGTK and GLib all you want for having
unsafe signal handling, I don't care.

  Regards.

On 9/4/06, Nick Maclaren [EMAIL PROTECTED] wrote:
 Jean-Paul Calderone [EMAIL PROTECTED] wrote:
 
  Thanks for expounding.  Given that it is basically impossible to do
  anything useful in a signal handler according to the relevant standards
  (does Python's current signal handler even avoid relying on undefined
  behavior?), how would you suggest addressing this issue?

 Much as you are doing, and I described, but the first step would be
 to find out what 'most' Python people need for signal handling in
 threaded programs.  This is because there is an unavoidable conflict
 between portability/reliability and functionality.

 I would definitely block all signals in threads, except for those that
 are likely to be generated ON the thread (SIGFPE etc.)  It is a very
 good idea not to touch the handling of several of those, because doing
 so can cause chaos.

 I would have at least two 'standard' handlers, one of which would simply
 set a flag and return, and the other of which would abort.  Now, NEITHER
 is a very useful specification, but providing ANY information is risky,
 which is why it is critical to know what people need.

 I would not TRUST the blocking of signals, so would set up handlers even
 when I blocked them, and would do the minimum fiddling in the main
 thread compatible with decent functionality.

 I would provide a call to test if the signal flag was set, and another
 to test and clear it.  This would be callable ONLY from the main thread,
 and that would be checked.

 It is possible to do better, but that starts needing serious research.

  It seems to me that it is actually possible to do useful things in a
  signal handler, so long as one accepts that doing so is relying on
  platform specific behavior.

 Unfortunately, that is wrong.  That was true under MVS and VMS, but
 in Unix and Microsoft systems, the problem is that the behaviour is
 both platform and circumstance-dependent.  What you can do reliably
 depends mostly on what is going on at the time.

 For example, on many Unix and Microsoft platforms, signals received
 while you are in the middle of certain functions or system calls, or
 certain particular signals (often SIGFPE), call the C handler with a
 bad set of global pointers or similar.  I believe that this is one of
 reasons (perhaps the main one) that some such failures so often cause
 debuggers to be unable to find the stack pointer.

 I have tracked a few of those down, and have occasionally identified
 the cause (and even got it fixed!), but it is a murderous task, and
 I know of few other people who have ever succeeded.

  How hard would it be to implement this for the platforms Python supports,
  rather than for a hypothetical standards-exact platform?

 I have seen this effect on OSF/1, IRIX, Solaris, Linux and versions
 of Microsoft Windows.  I have never used a modern BSD, haven't used
 HP-UX since release 9, and haven't used Microsoft systems seriously
 in years (though I did hang my new laptop in its GUI fairly easily).

 As I say, this isn't so much a platform issue as a circumstance one.


 Regards,
 Nick Maclaren,
 University of Cambridge Computing Service,
 New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
 Email:  [EMAIL PROTECTED]
 Tel.:  +44 1223 334761Fax:  +44 1223 334679
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

We have to resort to timeouts in pygtk in order to catch unix signals
in threaded mode.
The reason is this.  We call gtk_main() (mainloop function) which
blocks forever.  Suppose there are threads in the program; then any
thread can receive a signal (e.g. SIGINT).  Python catches the signal,
but doesn't do anything; it simply sets a flag in a global structure
and calls Py_AddPendingCall(), and I guess it expects someone to call
Py_MakePendingCalls().  However, the main thread is blocked calling a
C function and has no way of being notified it needs to give control
back to python to handle the signal.  Hence, we use a 100ms timeout
for polling.  Unfortunately, timeouts needlessly consume CPU time and
drain laptop batteries.

According to [1], all python needs to do to avoid this problem is
block all signals in all but the main thread; then we can guarantee
signal handlers are always called from the main thread, and pygtk
doesn't need a timeout.

Another alternative would be to add a new API like
Py_AddPendingCallNotification, which would let python notify
extensions that new pending calls exist and need to be processed.

  But I would really prefer the first alternative, as it could be
fixed within python 2.5; no need to wait for 2.6.

  Please, let's make Python ready for the enterprise! [2]

[1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3
[2] http://perkypants.org/blog/2006/09/02/rfte-python/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/2/06, Nick Maclaren [EMAIL PROTECTED] wrote:

 According to [1], all python needs to do to avoid this problem is
 block all signals in all but the main thread; then we can guarantee
 signal handlers are always called from the main thread, and pygtk
 doesn't need a timeout.

1) That page is password protected, so I can't see what it says, and
am disinclined to register myself to yet another such site.


 Oh, sorry, here's the comment:

(coment by Arjan van de Ven):
| afaik the kernel only sends signals to threads that don't have them blocked.
| If python doesn't want anyone but the main thread to get signals, it
should just
| block signals on all but the main thread and then by nature, all
signals will go
| to the main thread



2) No way, Jose, anyway.  The POSIX signal handling model was broken
beyond redemption, even before threading was added, and the combination
is evil almost beyond belief.  That procedure is good practice, yes,
but that is NOT all that you have to do - it may be all that you CAN
do, but that is not the same.

Nope.  Sorry, but you can't solve a broken design by adding interfaces.


 Well, Python has a broken design too; it postpones tasks and expects
to magically regain control in order to finish the job.  That often
doesn't happen!



   But I would really prefer the first alternative, as it could be
 fixed within python 2.5; no need to wait for 2.6.

It clearly should be done, assuming that Python's model is that it
doesn't want to get involved with subthread signalling (and I really,
but REALLY, recommend not doing so).  The best that can be done is to
say that all signal handling is the business of the main thread and
that, when the system bypasses that, all bets are off.


 Python is halfway there; it assumes signals are to be handled in the
main thread.  However, it _catches_ them in any thread, sets a flag,
and just waits for the next opportunity when it runs again in the main
thread.  It is precisely this split handling of signals that is
failing now.

 Anyway, attached a patch that should fix the problem in posix
threads systems, in case anyone wants to review.

 Cheers.
Index: Modules/threadmodule.c
===
--- Modules/threadmodule.c	(revision 51687)
+++ Modules/threadmodule.c	(working copy)
@@ -12,6 +12,17 @@
 
 #include pythread.h
 
+#if defined(_POSIX_THREADS)  defined(HAVE_PTHREAD_SIGMASK)  defined(HAVE_SIGNAL_H)
+#  include pthread.h
+#  include signal.h
+#  if defined(HAVE_BROKEN_PTHREAD_SIGMASK)
+#define SET_THREAD_SIGMASK sigprocmask
+#  else
+#define SET_THREAD_SIGMASK pthread_sigmask
+#  endif
+#endif
+
+
 static PyObject *ThreadError;
 
 
@@ -417,6 +428,15 @@
 	PyThreadState *tstate;
 	PyObject *res;
 
+  /* We want to block signals from all non-main threads, so
+   that we can guarantee they arrive always at the main
+   thread, where we can handle them */
+#ifdef SET_THREAD_SIGMASK
+sigset_t set;
+sigfillset(set);
+SET_THREAD_SIGMASK(SIG_BLOCK, set, NULL);
+#endif
+
 	tstate = PyThreadState_New(boot-interp);
 
 	PyEval_AcquireThread(tstate);
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyObject_CallFunction and 'N' format char

[Gustavo Carneiro]

On 6/25/06, Martin v. Löwis [EMAIL PROTECTED] wrote:
 Gustavo Carneiro wrote:
  However, PyObject_CallFunction does _not_
  consume such an object reference, contrary to what I believed for
  years.

 Why do you say that? It certainly does.

  Yes it does.  I could almost swear it didn't last night when I was
debugging this, but today it works as expected.  I guess I was just
sleepy... :P

  Sorry.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PyObject_CallFunction and 'N' format char

[Gustavo Carneiro]

  Sorry this is slightly offtopic, but I think it's important...

  According to recent experiments tracking down a memory leak, it
seems that PyObject_CallFunction(func, N, object) and
PyObject_CallFunction(func, O, object) are doing exactly the same
thing.  However, documentation says The C arguments are described
using a   Py_BuildValue() style format string..  And of course
Py_BuildValue consumes one object reference, according to the
documentation and practice.  However, PyObject_CallFunction does _not_
consume such an object reference, contrary to what I believed for
years.  God knows how many leaks I may have introduced in my
bindings... :|

  Any comments?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Shared libs on Linux (Was: test failures in test_ctypes (HEAD))

[Gustavo Carneiro]

On 5/3/06, Thomas Heller [EMAIL PROTECTED] wrote:
[Crossposting to both python-dev and ctypes-users, please respond to the listthat seems most appropriate]Guido van Rossum wrote: File /home/guido/projects/python/trunk/Lib/ctypes/__init__.py,
 line 288, in __init__ self._handle = _dlopen(self._name, mode) OSError: /usr/lib/libglut.so.3: undefined symbol: XGetExtensionVersionIt means that /usr/lib/libglut.so.3 is not explicitly linking to xlibs, but it should. Nothing wrong with python, it's the GL library that was not correctly linked.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dropping __init__.py requirement for subpackages

[Gustavo Carneiro]

On 4/26/06, Guido van Rossum [EMAIL PROTECTED] wrote:[...]
So I have a very simple proposal: keep the __init__.py requirement fortop-level pacakages, but drop it for subpackages. This should be asmall change. I'm hesitant to propose *anything* new for Python 2.5,so I'm proposing it for 
2.6; if Neal and Anthony think this would beokay to add to 2.5, they can do so. Damn these threads are so quick they are born and die off in 24 hours and don't give enough time for people to comment :(
 I'm a bit late in the thread, but I'm +1 (for 2.5) for the following reason. There are mainly two use cases for having a couple of modules foo.bar, and foo.zbr: 1. foo.bar and foo.zbr are both part of the same _package_, and are distributed together;
 2. foo.bar and foo.zbr are two independent modules, distributed separately, but which share a common 'foo' _namespace_, to denote affiliation with a project. The use case #1 is arguably more common, but use case #2 is also very relevant. It happens for a lot of GNOME python bindings, for example, where we used to have gnome, 
gnome.ui, gnome.vfs, gnome.applet, etc. Now the problem. Suppose you have the source package python-foo-bar, which installs $pythondir/foo/__init__.py and $pythondir/foo/bar.py. This would make a module called 
foo.bar available. Likewise, you can have the source package python-foo-zbr, which installs
$pythondir/foo/__init__.py and $pythondir/foo/zbr.py. This would make
a module called foo.zbr available. The two packages above install the file $pythondir/foo/__init__.py. If one of them adds some content to __init__.py, the other one will overwrite it. Packaging these two packages for 
e.g. debian would be extremely difficult, because no two .deb packages are allowed to intall the same file. One solution is to generate the __init__.py file with post-install hooks and shell scripts. Another solution would be for example to have only python-foo-bar install the __init__.py file, but then python-foo-zbr would have to depend on python-foo-bar, while they're not really related.
 I hope I made the problem clear enough, and I hope people find this a compelling argument in favour of eliminating the need for __init__.py. Regards,Gustavo Carneiro.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dropping __init__.py requirement for subpackages

[Gustavo Carneiro]

On 4/27/06, Phillip J. Eby [EMAIL PROTECTED] wrote:
At 03:48 PM 4/27/2006 +0200, Bernhard Herzog wrote:Gustavo Carneiro [EMAIL PROTECTED] writes:  Now the problem.Suppose you have the source package python-foo-bar,
  which installs $pythondir/foo/__init__.py and $pythondir/foo/bar.py.This  would make a module called foo.bar available.Likewise, you can have the  source package python-foo-zbr, which installs
 $pythondir/foo/__init__.py and  $pythondir/foo/zbr.py.This would make a module called foo.zbr available.   The two packages above install the file $pythondir/foo/__init__.py.If
  one of them adds some content to __init__.py, the other one will overwrite  it.Packaging these two packages for e.g. debian would be extremely  difficult, because no two .deb packages are allowed to intall the same
 file.   One solution is to generate the __init__.py file with post-install hooks  and shell scripts.Another solution would be for example to have only  python-foo-bar install the __init__.py file, but then python-foo-zbr would
  have to depend on python-foo-bar, while they're not really related.Yet another solution would be to put foo/__init__.py into a thirdpackage, e.g. python-foo, on which both python-foo-bar and
python-foo-zbr depend. You can't be serious. One package just to install a __init__.py file?
Or you can package them with setuptools, and declare foo to be a namespacepackage. Let's not assume setuptools are always used, shall we?
If installing in the mode used for building RPMs and debs, therewill be no __init__.py.Instead, each installs a .pth file that ensures adummy package object is created in sys.modules with an appropriate__path__.This solution is packaging-system agnostic and doesn't require
any special support from the packaging tool. I don't understand this solution. How can a .pth file create a 'dummy package'? Remember that the objective is to have foo.bar and 
foo.zbr modules, not just bar and zbr. But in any case, it already sounds like a convoluted solution. No way it can beat the obvious/simple solution: to remove the need to have __init__.py in the first place.
(The downside, however, is that neither foo.bar nor foo.zbr's __init__.pywill be allowed to have any content, since in some installation scenarios
there will be no __init__.py at all.) That's ok in the context of this proposal (not having __init__.py at all). 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dropping __init__.py requirement for subpackages

[Gustavo Carneiro]

On 4/27/06, Thomas Wouters [EMAIL PROTECTED] wrote:
On 4/27/06, Gustavo Carneiro 
[EMAIL PROTECTED] wrote:
On 4/27/06, Phillip J. Eby 

[EMAIL PROTECTED] wrote:

At 03:48 PM 4/27/2006 +0200, Bernhard Herzog wrote:Gustavo Carneiro [EMAIL PROTECTED]
 writes:  Now the problem.Suppose you have the source package python-foo-bar,
  which installs $pythondir/foo/__init__.py and $pythondir/foo/bar.py.This  would make a module called foo.bar available.Likewise, you can have the  source package python-foo-zbr, which installs
 $pythondir/foo/__init__.py and  $pythondir/foo/zbr.py.This would make a module called foo.zbr available.   The two packages above install the file $pythondir/foo/__init__.py.If
  one of them adds some content to __init__.py, the other one will overwrite  it.Packaging these two packages for e.g. debian would be extremely  difficult, because no two .deb packages are allowed to intall the same file.
 Yet another solution would be to put foo/__init__.py into a thirdpackage, e.g. python-foo, on which both python-foo-bar and
python-foo-zbr depend. You can't be serious. One package just to install a __init__.py file?
Sure. Have you counted the number of 'empty' packages in Debian lately? Sure. That is already a problem; let's not make it a worse problem for no good reason; I haven't heard a good reason to keep the __init__.py file besides backward compatibility concerns.
  Besides, Guido's original proposal is not a fix for your problem, either; he only proposes to change the requirement for *sub*packages.
 It *is* a solution for my problem. I don't need the __init__.py file for anything, since I don't need anything defined in the the 'foo' namespace, only the subpackages foo.bar and foo.zbr
.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dropping __init__.py requirement for subpackages

[Gustavo Carneiro]

On 4/27/06, Thomas Wouters [EMAIL PROTECTED] wrote:
On 4/27/06, Gustavo Carneiro 
[EMAIL PROTECTED] wrote:

 Besides, Guido's original proposal is not a fix for your problem, either; he only proposes to change the requirement for *sub*packages.
 It *is* a solution for my problem. I don't need the __init__.py file for anything, since I don't need anything defined in the the 'foo' namespace, only the subpackages 
foo.bar and foo.zbr
 No. Guido's original proposal is not a fix for your problem, because *it doesn't affect the 'foo' namespace*. Guido's original proposal still requires foo/__init__.py for your namespace to work, it just makes foo/bar/__init__.py and foo/zbr/__init__.py optional.
 Damn, you're right, I confused subpackage with submodule :P In that case, can I counter-propose to stop requiring the __init__.py file in [foo/__init__.py, foo/bar.py] ? ;-)
 The proposal would mean that a directory 'foo' with a single file bar.py would make the module 'foo.bar' available if the parent directory of 'foo' is in sys.path./me faces the pitchforks.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] adding Construct to the standard library?

[Gustavo Carneiro]

why include Construct?* the struct module is very nice, but very limited and non-pythonic as well
* pure python (no platform/security issues) IMHO this is a drawback. More on this below.
* lots of people need to parse and build binary data structures, it's not an esoteric library
* license: public domain* quite a large user base for such a short time (proves the need of the community) Indeed, I wish I had known about this a year ago; it would have saved me a lot of work. Of course it probably didn't exist a year ago... :(
* easy to use and extend (follows the componentization pattern)
* declarative: you don't need to write executable code for most cases
 Well, declarative is less flexible. OTOH declarative is nice in the way it is more readable and allows more optimisations.
why not:* the code is (very) young. stable and all, but less than a month on the loose. * new features may still be added / existing ones may be changed in a non-backwards-compatible manner
so why am i saying this now, instead of waiting a few months for it to maturet? 
well, i wanted to get feedback.
those of you who have seen/used the library, please tell me what you think:* is it suitable for a standard library?
* what more features would you want?* any changes you think are necessary? This is a very nice library indeed. But the number one feature that I need in something like this would be to use C. That's because of my application specific requirements, where i have observed that reapeatedly using 
struct.pack/unpack and reading bytes from a stream represents a considerable CPU overhead, whereas the same thing in C would be ultra fast. IMHO, at least in theory Construct could have small but fast C extension to take care of the encoding and decoding, which is the critical path. Everything else, like the declaration part, can be python, as it is usually done once on application startup.
 If you agree to go down this path I might even be able to volunteer some of my time to help, but it's not my decision. Best regards.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Class decorators

[Gustavo Carneiro]

On 3/29/06, Phillip J. Eby [EMAIL PROTECTED] wrote:
At 11:35 PM 3/28/2006 -0500, Fred L. Drake, Jr. wrote:For Zope 3, we have decorators that work with the component architecture (I'msure Phillip is familiar with these).They're used with functions toindicate that the function adapts a particular kind of object, or that it
implements or provides a particular interface.We have different functionsthat get used for this purpose in classes that are executed within the bodyof the class.There's some merit to being able to use a single set of
functions in both cases, since the use cases are the same.I'm not sure I'dwant to change the existing pattern, though, since it's already so widespreadwithin the Zope 3 codebase (including 3rd-party components).
If we're using Zope 3 as an example, I personally find that: class Foo: Docstring here, blah blah blah  implements(IFoo)is easier to read than:
 @implements(IFoo) class Foo: Docstring here, blah blah blah  Yeah, but in the first case  implements(IFoo) has to do dirty hacks involving 
sys.getframe(), so you win in once place but lose in the other one.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com