date:20140406

On Sun, Apr 6, 2014 at 4:29 PM, Anthony Papillion papill...@gmail.com wrote:
 No particular reason at all. I've Bern dabbling in Python for the last bit
 and am just writing code based on the samples or examples I'm finding.  What
 was the tipoff that this was not Python 3? Would there be a large difference
 in this code if it was Python 3?

The tip-off was that you have no parentheses around print's arguments.
Behold the vast difference that told me which it was:

# Python 2: print is a statement
print int(row['YEAR'])

# Python 3: print is a function
print(int(row['YEAR']))

So incredibly different :) But it's enough to show that you're on Python 2.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I parse this correctly?

2014-04-06 Thread Ben Finney

Anthony Papillion papill...@gmail.com writes:

 On Apr 5, 2014, at 23:21, Ben Finney ben+pyt...@benfinney.id.au wrote:
  Alternatively, if you just want to do integer arithmetic on the
  year, you don't need the ‘datetime’ module at all.

 True. But I do actually need to some date based calculations.
 Basically I'm processing a large data set and calculating time
 intervals between entries

Okay. So, it seems that some entries have (some of?) the date components
blank.

There is no sensible general-purpose default for ‘datetime.date’, so you
will need to decide what “empty date” means for your data.

  Python doesn't have “cast”; instead, you request the creation of a
  new object by calling the type.

 Hmm, interesting. I need to think on that for a moment.  I may well
 have completely misunderstood a major part of Python all this time.

Yes, it's important to recognise that all Python's built-in types are
callable just as user-defined types are; and by calling them, you are
requesting a new instance.

  print int(row['YEAR'])
 
  What do you expect this to return when ‘row['YEAR']’ is ‘’ (empty
  string)?

 I expected a defaut value to be returned, perhaps 0.

You'll need to be aware that the Gregorian calendar (which is what
‘datetime.date’ uses) has no year zero. 1 BCE is immediately succeeded
by 1 CE.

URL:https://en.wikipedia.org/wiki/Year_Zero

So, again, there's no good general-purpose default year in our calendar.
Any system will need an explicit decision for the most sensible default
for its purpose.

 I see now from another response that this is not the case and so I've
 fixed it to read

 print int(row['YEAR'] or )

“foo or bar” is not a Pythonic way to get a default value; it relies on
quirks of implementation and is not expressive as to the meaning you
intend.

Rather, be explicit:

# Default to the beginning of the project.
year = 1969
if row['YEAR']:
# Use the value as a text representation of a year.
year = int(row['YEAR'])

Also, be aware that Python allows leading-zero integer literals, but
URL:http://catb.org/jargon/html/H/hysterical-reasons.html interprets
them not as decimal (base ten), but as octal (base eight).

If “this integer is base-eight for a good reason explained nearby” is
not your intent, don't put a leading-zero integer literal in the code.

 Thank you! I actually like the fact that [Python's ‘int’ initialiser]
 won't simply fill something in. It makes things more predictable and
 stable.

Welcome to a dependable language :-)

-- 
 \   “Firmness in decision is often merely a form of stupidity. It |
  `\indicates an inability to think the same thing out twice.” |
_o__)—Henry L. Mencken |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Keeping track of things with dictionaries

2014-04-06 Thread Peter Otten

Giuliano Bertoletti wrote:

 I frequently use this pattern to keep track of incoming data (for
 example, to sum up sales of a specific brand):
 
 =
 
 # read a brand record from a db
 ...
 
 # keep track of brands seen
 obj = brands_seen.get(brandname)
 if obj is None:
 obj = Brand()
 brands_seen[brandname] = obj
 
 obj.AddData(...)  # this might for example keep track of sales
 
 =
 
 as you might guess, brands_seen is a dictionary whose keys are
 brandnames and whose values are brand objects.
 
 Now the point is: is there a cleverer way to do this?
 
 Basically what I'm doing is query the dictionary twice if the object
 does not exist.
 
 What I would like to understand is if there's some language built-in
 logic to:
 
 - supply a function which is meant to return a new object
 - have the interpreter to locate the point in the dictionary where the
 key is to be
 - if the key is already there, it returns the value/object associated
 and stops
 - if the key is not there, it calls the supplied function, assigns the
 returned value to the dictionary and return the object.

Cudos, you give a precise discription of your problem in both english and 
code.

There is a data structure in the stdlib that fits your task. With a 
collections.defaultdict your code becomes

from collections import defaultdict

brands_seen = defaultdict(Brand)
brands_seen[brandname].add_data(...) # Method name adjusted to PEP 8

Side note: If you needed the key in the construction of the value you would 
have to subclass

class BrandsSeen(dict):
def __missing__(self, brandname):
result = self[brandname] = Brand(brandname)
return result

brands_seen = BrandsSeen()
brands_seen[brandname].add_data(...)


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Mutable objects inside tuples - good or bad?

On Sun, Apr 6, 2014 at 5:55 PM, Devin Jeanpierre jeanpierr...@gmail.com wrote:
 On Sun, Apr 6, 2014 at 12:25 AM, Gary Herron
 gary.her...@islandtraining.com wrote:
 On 04/05/2014 11:53 PM, John Ladasky wrote:

 I find this programming pattern to be useful... but can it cause problems?

 No.

 What kind of problems are you considering?  It won't break Python. It's
 perfectly legal code.

 Agreed. Putting mutable objects inside tuples is common and totally OK.

There are many programming habits that can cause problems, even though
they won't break Python and are legal code. :)

 The tuple c is still immutable, consisting of two specific objects, and (as
 always) without regard to the specifics or contents of those two objects.

 You can choose to define mutability that way, but in many contexts
 you'll find that definition not very useful.

 c is such that you could have another variable d, where the following
 interpreter session fragment is easily possible:

 c == d
 True
 foo(c)
 c == d
 False

What you're looking at here is hashability, not mutability. Compare:

 a = (1,2,3)
 hash(a)
2528502973977326415
 b = ([1],[2],[3])
 hash(b)
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: unhashable type: 'list'

Both tuples are immutable, but the former is hashable because all its
members are hashable, while the latter is not. You can't trust that
equality with b is constant:

 c = ([1],[2],[3])
 b == c
True
 b[2][0]=4
 b == c
False

Going back to the original question, though: I do not believe that
putting mutable objects inside tuples is a problem. I've done it
frequently myself, and it's never caused confusion. So go right ahead,
do it if it makes sense!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I parse this correctly?

On Sun, Apr 6, 2014 at 6:16 PM, Ben Finney ben+pyt...@benfinney.id.au wrote:
 print int(row['YEAR'] or )

 “foo or bar” is not a Pythonic way to get a default value; it relies on
 quirks of implementation and is not expressive as to the meaning you
 intend.

 Rather, be explicit:

 # Default to the beginning of the project.
 year = 1969
 if row['YEAR']:
 # Use the value as a text representation of a year.
 year = int(row['YEAR'])

What's wrong with foo or bar as a means of defaulting a blank
string? (Obviously this assumes you know you have a string, as it'll
equally default a 0 or a 0.0 or a [] or anything else false.) The
definition of the or operator is that it returns its first argument
if it's true, otherwise it returns its second argument. That's not a
quirk of implementation. This exact example (with strings, no less!)
can be found in help(or) and in the docs:

https://docs.python.org/3/reference/expressions.html#boolean-operations

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A data conversion question

2014-04-06 Thread Peter Otten

Mok-Kong Shen wrote:

 A newbie's question of curiosity:
 
 If I have
 
 g=[1,[2]] and
 
 bg=bytearray(str(g),latin-1)
 
 could I somehow get back from bg a list g1 that is the same as g?

Not for arbitrary values, but for lists, ints, and a few other types that's 
not a problem:

 g = [1, [2]]
 bg = bytearray(str(g), latin-1)
 bg
bytearray(b'[1, [2]]')
 import ast
 ast.literal_eval(bg.decode(latin-1))
[1, [2]]

See also https://docs.python.org/dev/library/ast.html#ast.literal_eval

Note that while eval() instead of ast.literal_eval() would also work you 
should avoid it. eval() can execute arbitrary Python code and is thus a big 
security whole when applied to user-provided data.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

Mark H Harris harrismh...@gmail.com:

 On 4/4/14 4:53 AM, Steven D'Aprano wrote:
 Python is not a computer-science-ey language.

 Every programming language is interesting from a comp sci standpoint.
 Some are more useful for research; python is one of those.

 For what reasons do you disagree?

Computer science doesn't mean anything related to computers.
Physicists typically couldn't care less about your heating up your lunch
in the microwave oven. Similarly, computer scientists aren't interested
in the mundane applications of their lofty research topics.

Python, BTW, is perfectly suitable for computer science. Normally,
though, you either use a pseudolanguage or some sort of formalism that
hasn't been implemented.

In theoretical computer science, they cherish off-the-wall models that
detach the topic from everyday applications. Here are examples that I
remember from graduate school:

 * combinatory birds in forests

 * unfaithful husbands on an island ruled by female logicians

 * dining philosophers getting into a deadlock over forks

 * Byzantine generals trying to agree on a surprise onslaught on a
   besieged city


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Mutable objects inside tuples - good or bad?

2014-04-06 Thread Devin Jeanpierre

On Sun, Apr 6, 2014 at 1:25 AM, Chris Angelico ros...@gmail.com wrote:
 On Sun, Apr 6, 2014 at 5:55 PM, Devin Jeanpierre jeanpierr...@gmail.com 
 wrote:
 Agreed. Putting mutable objects inside tuples is common and totally OK.

 There are many programming habits that can cause problems, even though
 they won't break Python and are legal code. :)

Yes, but this isn't one of them.

 c is such that you could have another variable d, where the following
 interpreter session fragment is easily possible:

 c == d
 True
 foo(c)
 c == d
 False

 What you're looking at here is hashability, not mutability. Compare:

No, that is not what I am looking at. There are hashable objects with
the property I described, and unhashable objects without it.

My point in that example was that sometimes it is more useful to talk
about entire objects and their behavior as a whole. Calling the object
immutable when it has mutable state is misleading in this context.

-- Devin
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I parse this correctly?


On 06/04/2014 09:17, Chris Angelico wrote:

On Sun, Apr 6, 2014 at 4:29 PM, Anthony Papillion papill...@gmail.com wrote:

No particular reason at all. I've Bern dabbling in Python for the last bit
and am just writing code based on the samples or examples I'm finding.  What
was the tipoff that this was not Python 3? Would there be a large difference
in this code if it was Python 3?


The tip-off was that you have no parentheses around print's arguments.
Behold the vast difference that told me which it was:

# Python 2: print is a statement
print int(row['YEAR'])

# Python 3: print is a function
print(int(row['YEAR']))

So incredibly different :) But it's enough to show that you're on Python 2.

ChrisA



I'd recommend using this import statement in Python 2 so you get used to 
print being a function.


from __future__ import print_function

--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com


--
https://mail.python.org/mailman/listinfo/python-list

Re: Python streaming media server


On 06/04/2014 05:31, Wesley wrote:

在 2014年4月5日星期六UTC+8下午6时11分02秒，Wesley写道：

Hi,

   Anyone knows open source streaming media server written by Python?



I am trying to setup a streaming media server in python, wanna find an existing 
one and have a look.



Thanks.

Wesley


After a lot google work, I am looking at Flumotion.
Need to check the streaming mode and file formats it support.

Wesley



Would you please use the mailing list 
https://mail.python.org/mailman/listinfo/python-list or read and action 
this https://wiki.python.org/moin/GoogleGroupsPython to prevent us 
seeing double line spacing and single line paragraphs, thanks.


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com


--
https://mail.python.org/mailman/listinfo/python-list

Re: How can I parse this correctly?

On Sun, Apr 6, 2014 at 9:32 PM, Mark Lawrence breamore...@yahoo.co.uk wrote:
 I'd recommend using this import statement in Python 2 so you get used to
 print being a function.

 from __future__ import print_function

Or better still, just write Python 3 code - then you get to take
advantage of all the fancy new features :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I parse this correctly?


On 06/04/2014 12:54, Chris Angelico wrote:

On Sun, Apr 6, 2014 at 9:32 PM, Mark Lawrence breamore...@yahoo.co.uk wrote:

I'd recommend using this import statement in Python 2 so you get used to
print being a function.

from __future__ import print_function


Or better still, just write Python 3 code - then you get to take
advantage of all the fancy new features :)

ChrisA



I meant to say, If you're stuck with Python 2 ... :(

--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com


--
https://mail.python.org/mailman/listinfo/python-list

Re: Python streaming media server

2014-04-06 Thread Sturla Molden

Wesley nisp...@gmail.com wrote:

 Not open source, but there is a famous closed-source one called YouTube.
 
 Are you kidding?
 I know youtube, but do you think we can use it setup our own streaming media 
 server?

Obviously not. 

Before YouTube was bought by Google, it was common knowledge that it ran on
Stackless Python. So a streaming media server on Python is absolutely
possible. But no, I don't know of one you can set up and use on your own.

You can make a highly scalable server with PyZMQ and Tornado or Twisted.
NumPy is great for storing binary data like media streams. HDF5 (PyTables
or h5py) might be a better database than some SQL server, as it is capable
of highly scalable parallel binary i/o.  

Sturla

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I parse this correctly?

2014-04-06 Thread Tim Chase

On 2014-04-06 14:21, Ben Finney wrote:
 I assume you mean you will be creating ‘datetime.date’ objects. What
 will you set as the month and day?
 
 Alternatively, if you just want to do integer arithmetic on the
 year, you don't need the ‘datetime’ module at all.

Even if you do the arithmetic by hand, it's still nice to use the
datetime module to parse for sane dates:

  year = 2004
  month = 2
  day = 29

what should month  day be if you increment/decrement the year by
one?  The datetime module will throw a ValueError which is a nice
check for a valid date.  I've had to do things like this in a loop to
sanitize dates (depending on which field is being inc/dec'ed, by how
much, and which direction it's going) and it's nice to just have a

  y,m,d = initial = some_date.timetuple()[:3] #
  result = None
  while result is None:
y,m,d = twiddle(y, m, d)
try:
  result = datetime(y, m, d)
except ValueError:
  result = None
  log.info(Shifted %r - %r, initial, result)

where twiddle() is whatever business logic I need for this particular
case.  For me usually, it's adjusting by one month for billing
purposes.

-tkc


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Mutable objects inside tuples - good or bad?

On Sunday, April 6, 2014 1:40:58 PM UTC+5:45, Devin Jeanpierre wrote:
 
 You can choose to define mutability that way, but in many contexts
 you'll find that definition not very useful.
 
 c is such that you could have another variable d, where the following
 interpreter session fragment is easily possible:
 
 
  c == d
 True
  foo(c)
  c == d
 False

Its called referential transparency (or rather the lack of it)
And that is why it (Johns question) is not a good idea.

In general worshipping overzealously at the altar of RT produces functional
programming. To the non-zealots this has the characteristic of 
Throw out baby with bathwater

On the other hand imperative programming is a source of more problems than
people realize:
http://blog.languager.org/2012/11/imperative-programming-lessons-not.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

2014-04-06 Thread Steven D'Aprano

On Sun, 06 Apr 2014 12:05:16 +0300, Marko Rauhamaa wrote:

 Mark H Harris harrismh...@gmail.com:
 
 On 4/4/14 4:53 AM, Steven D'Aprano wrote:
 Python is not a computer-science-ey language.

 Every programming language is interesting from a comp sci standpoint.
 Some are more useful for research; python is one of those.

 For what reasons do you disagree?
 
 Computer science doesn't mean anything related to computers.
 Physicists typically couldn't care less about your heating up your lunch
 in the microwave oven. Similarly, computer scientists aren't interested
 in the mundane applications of their lofty research topics.
 
 Python, BTW, is perfectly suitable for computer science. 

I don't think it is. Python is not a pure functional language, so it's 
very difficult to prove anything about the code apart from running it. 
For example, see Brett Cannon's master's thesis, where he essentially 
demonstrates that:

- you can't do compile-time type inference of general types in Python;

- although you can infer a very small amount of information about a 
  few built-in types;

- adding specialized byte-codes to handle those types gives, at best,
  a marginal performance boost, and sometimes even slows code down.


To quote from the conclusion:

   Introducing over 3,000 lines of new C code to Python’s compiler 
to get, at best, a 1% improvement is in no way justified. The 
level of added complexity that would be introduced into the
compilation step would definitely need to lead to a noticeable
performance improvement, the 5% that was the goal, to justify the
cost of code maintenance.

http://citeseerx.ist.psu.edu/viewdoc/download?
doi=10.1.1.90.3231rep=rep1type=pdf


What does it mean to say that a language like Python is suitable for use 
in computer science? It can mean (at least) four things:

(1) If you do an undergraduate computer science course, they will teach 
you Python.

While this is good for the general Python community, it's hardly *doing* 
computer science. It's *learning* computer science. (I read a book by 
Richard Dawkins. That doesn't mean I'm a biologist.) So while I agree 
that it is significant that some universities will teach Python to 
undergraduates, I don't count this as Python being used in computer 
science. I think we need to look at postgraduate use of Python, not 
undergraduate.

(2) Some post-grads use Python as a tool, e.g. they use a Python script 
to analyse some data. In this case, the use of Python is incidental to 
the research they are doing. They might have used Perl, or a bash script, 
or calculated the results by hand. In a similar fashion, they probably 
wrote up their results using Microsoft Word. It's just a tool.

(3) Some post-grads do original research *into* Python, as a language. 
Brett Cannon's thesis is proof that this has happened at least once.

I think this does count as doing computer science with Python, although 
only marginally. No slight intended, but it should be obvious that 
something like Brett's thesis has very little application outside of 
Python itself. Perhaps a little: if there is another language with 
similar types of dynamism as Python, you might conclude that it too is 
not a good candidate for compile-time type inference.

(4) This is the category which I was referring to when I said that Python 
wasn't a computer-science-ey language: do people use Python for 
research into language-independent fundamental principles of computation? 
I don't think so. I agree with Marko that normally you:

 you either use a pseudolanguage or some sort of formalism that
 hasn't been implemented.

E.g. most of the really deep stuff by Turing wasn't even performed on a 
computer, since there were no computers[1], or languages to program them 
in. A lot (all?) of Knuth's published work is written in an assembly 
language for an imaginary processor. Douglas Hofstadter invented two 
languages, BlooP and FlooP, to demonstrate the difference between 
programming languages that are, or aren't, Turing complete. (He also 
named a mythical super-Turing language GlooP.)

Some languages are better suited for academic research of this nature. 
Python is too... messy, I suppose. Python's mix of imperative, functional 
and OOP paradigms makes it really useful for solving problems, but less 
useful for academic research of this type, where pure functional, pure 
OOP paradigms are more useful. Naturally I'm not saying that there is 
*absolutely no* comp-sci work done using Python, that would be foolish, 
only that it is in a minority and is not well-suited for the sort of 
problems academics are interested in.

But since I'm not a computer scientist, perhaps I'm wrong. Anyone have 
any studies showing what percentage of research papers use various 
languages?


 In theoretical computer science, they cherish off-the-wall models that
 detach the topic from everyday applications. Here are examples that I
 remember from graduate school:
 
  * combinatory

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

On Mon, Apr 7, 2014 at 2:52 AM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
 (4) This is the category which I was referring to when I said that Python
 wasn't a computer-science-ey language: do people use Python for
 research into language-independent fundamental principles of computation?
 I don't think so. I agree with Marko that normally you:

 you either use a pseudolanguage or some sort of formalism that
 hasn't been implemented.

 E.g. most of the really deep stuff by Turing wasn't even performed on a
 computer...

A simple reason for that is summed up in the Zen of Python:
Practicality beats purity. For a comp sci theoretical study, you want
something that exemplifies purity. That's why you get examples like
the ones mentioned below - a dining philosopher is fundamentally
unable to do such a simple thing as look to see what his neighbours
are doing, and is also apparently unable to think and eat at the same
time (plus, why on earth can't they afford a few more forks in the
interests of hygiene??!?). Defining all of mathematics in terms of
lambda is wonderfully pure, and utterly useless for any practical
purpose.

It's the same, in my opinion, with the eternal arguments about
functional vs imperative vs declarative programming languages, and
with the differences between compilers and interpreters, and whether
something's a second-generation or third-generation or
fourth-generation language. You can define all those terms in nice
pure ways (a compiler translates code into something that can be
executed directly, but an interpreter parses code bit by bit and
executes it), but most actually-usable systems blur the lines. I
still haven't seen any real definition of FP or OOP (especially the
latter, O!) that doesn't ultimately come down to avoid these language
features which violate FP/OOP principles, which means that most
programming languages (and more so with popular languages) are neither
and/or both. It's all very well to say that everything is a function
whose return value depends solely on its arguments, but practicality
demands that you allow side effects in certain places. And it's all
very well to say that everything's an object and every bit of code is
a method, but aiming too far for purity results in Java-like syntactic
salt. Pike avoids this somewhat by letting you pretend that it's a
C-like module with top-level functions, but actually it's instantiated
an object and called a method on it. That's occasionally useful, but
most of the time you just ignore it and work imperatively. Python goes
a bit further: top-level is just code like anything else, and it gets
executed straight down the line. Practicality beats purity.

  * unfaithful husbands on an island ruled by female logicians

 I don't know that one.

Me neither, although I can see elements of classic logic analysis
elements. Islands ruled by logicians, people who always tell the truth
/ always tell exact falsehoods, etc, etc. I don't know of any that
involve unfaithful husbands, but it wouldn't surprise me. Would like
to hear it though.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

On Sunday, April 6, 2014 10:22:21 PM UTC+5:30, Steven D'Aprano wrote:
 On Sun, 06 Apr 2014 12:05:16 +0300, Marko Rauhamaa wrote:

  Mark H Harris :
  On 4/4/14 4:53 AM, Steven D'Aprano wrote:
  Python is not a computer-science-ey language.
  Every programming language is interesting from a comp sci standpoint.
  Some are more useful for research; python is one of those.
  For what reasons do you disagree?
  Computer science doesn't mean anything related to computers.
  Physicists typically couldn't care less about your heating up your lunch
  in the microwave oven. Similarly, computer scientists aren't interested
  in the mundane applications of their lofty research topics.
  Python, BTW, is perfectly suitable for computer science. 

 I don't think it is. Python is not a pure functional language, so it's 
 very difficult to prove anything about the code apart from running it. 
 For example, see Brett Cannon's master's thesis, where he essentially 
 demonstrates that:

 - you can't do compile-time type inference of general types in Python;

 - although you can infer a very small amount of information about a 
   few built-in types;

 - adding specialized byte-codes to handle those types gives, at best,
   a marginal performance boost, and sometimes even slows code down.

 To quote from the conclusion:

Introducing over 3,000 lines of new C code to Python's compiler 
 to get, at best, a 1% improvement is in no way justified. The 
 level of added complexity that would be introduced into the
 compilation step would definitely need to lead to a noticeable
 performance improvement, the 5% that was the goal, to justify the
 cost of code maintenance.

 http://citeseerx.ist.psu.edu/viewdoc/download?
 doi=10.1.1.90.3231rep=rep1type=pdf

Thanks for the link.

It think however you are reading it more widely than intended.
The appropriate context is a few paras below:

| In the end, it seems that Python, as a language, is not geared towards
| type inference. Its flexibility, considered a great strength,
| interferes with how extensive type inference can be performed. Without
| a significant change to the language, type inference is not worth the
| hassle of implementation

So...
Yes if CS begins and ends with type-inference then your conclusion would be 
valid. However consider that some of the things that people did around 40 years
ago and do today

- use FORTRAN for numerical/simulation work --  now use scipy/sage etc
- NLP with Lisp/Prolog -- look at Nltk
- ??? with Data Analysis in Pandas
- Scheme (basis for programming pedagogy, semantics research) - Python

you can add/multiply ad libitum

Yeah you covered this in your (2) as ...just a tool...

Ask some recent PhD about what is for you just an almost irrelevant
tool and you are very likely to find that that choice may well have
been the difference between completing the research and giving up.

I think python wins because it (usually) lets people do their thing
(includes but not limited to CS-research)
and gets out of the way.  To say therefore that it is irrelevant to the 
research is a strange inversion of its advantages.

[Or simply just switch to C++ for 3 months and report back with
the increment in your white-hair-count]


In short, I just dispute your 'just'!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Yet Another Switch-Case Syntax Proposal

2014-04-06 Thread Marco S.

On 3 April 2014 20:12, Ian Kelly ian.g.kelly-at-gmail.com |
python-list@python.org| dnl5yyr...@sneakemail.com wrote:
 Use this instead [of continue]:

 switch day case in (Mon, Tue, Wed, Thu, Fri):
 go_to_work = True
 day_type = ferial
 if day in (Tue, Thu):
 lunch_time = datetime.time(11, 30)
 meeting_time = datetime.time(12, 30)
 else:
 lunch_time = datetime.time(12)
 meeting_time = datetime.time(14)
 case in (Sat, Sun):
 go_to_work = False
 day_type = festive

 You get an extra level of indentation this way, but it reads less like
 spaghetti code.


Well, if you have to add an if-else to your switch-case, it means
switch-case syntax is not so useful.
An alternative is to imitate elif, so you'll have elcase. This way we don't
need continue. Since I do not like elcasein, the best solution is to do as
suggested by many of you, so case in instead of casein.
But if you can write case in, why you can't write case comp_operator in
general? With this syntax you can do also case is not, case  etc... Your
example will turn into

briefing_days = (Tue, Thu)
festive_days = (Sat, Sun)

switch day case in briefing_days:
lunch_time = datetime.time(11, 30)
meeting_time = datetime.time(12, 30)
case not in briefing_days + festive_days:
lunch_time = datetime.time(12)
meeting_time = datetime.time(14)
case in festive_days:
go_to_work = False
day_type = festive
else:
go_to_work = True
day_type = ferial

The if-else equivalent will be:

if day in briefing_days:
lunch_time = datetime.time(11, 30)
meeting_time = datetime.time(12, 30)
if day not in briefing_days + festive_days:
lunch_time = datetime.time(12)
meeting_time = datetime.time(14)
if day in festive_days:
go_to_work = False
day_type = festive
else:
go_to_work = True
day_type = ferial


If you don't specify comp_operator, the default is ==. Example:

switch day_num case 1:
day_str = Monday
elcase 2:
day_str = Thursday
else:
day_str = etcetera
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

On Mon, Apr 7, 2014 at 3:31 AM, Rustom Mody rustompm...@gmail.com wrote:
 However consider that some of the things that people did around 40 years
 ago and do today

 - use FORTRAN for numerical/simulation work --  now use scipy/sage etc
 - NLP with Lisp/Prolog -- look at Nltk
 - ??? with Data Analysis in Pandas
 - Scheme (basis for programming pedagogy, semantics research) - Python

 you can add/multiply ad libitum

 Yeah you covered this in your (2) as ...just a tool...

 Ask some recent PhD about what is for you just an almost irrelevant
 tool and you are very likely to find that that choice may well have
 been the difference between completing the research and giving up.

 I think python wins because it (usually) lets people do their thing

Allow me to put it another way. Mathematicians use the language of
algebra to describe their research; they don't, by and large, use a
programming language. They use pencils and paper [1] as tools to get
their work done, and may well have strong opinions on which pencil
brand is the best, but the point of the pencil (pun intended) is to
enable something else. It's supposed to get out of the way and let
them do their thing. Python is highly practical because it gets out of
the way. It's not the way that you develop programming language
theory, though.

I might start out designing a language with the express purpose of
implementing everything as an expression. The whole program consists
of one long expression, with perhaps the semicolon being a sequence
point that evaluates its left side, then evaluates its right side, and
returns the latter (like the C comma operator). I could then go
through a whole lot of lovely mental exploration as to what the
benefits and costs of that system are, all the while writing nothing
more than design documents and example code. At some point, if I'm
happy with it, I'll write a reference implementation, and maybe then
I'll use Python for the job. But that's not using Python to explore a
language concept; that's using Python to rapidly prototype the
executable code that I need in order to show my new language at work.
All the work of developing the language is done in the design stage,
with nothing at all even needing a computer (although I *guarantee*
you that if I were to start something like that, I'd find part way
through that I've made some fundamental mistakes early on - and a
computer is better for editing text than anything on paper). I could
just as easily write my reference implementation using yacc/bison and
C, and it wouldn't be materially different.

Using Python at the design stage would be what Steven's talking about
- actually using it to build the theory of programming. I have about
as much experience in the area as he has, so we can't speak to the
lack of examples, but that's the sort of example it would take.

ChrisA

[1] As the old joke goes: The physics department needs a whole lot of
expensive equipment, but the math department needs only pencils,
paper, and wastepaper baskets. And the philosophy department goes even
further: they don't need wastepaper baskets.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)


On 06/04/2014 18:27, Chris Angelico wrote:

(plus, why on earth can't they afford a few more forks in the
interests of hygiene??!?).


They couldn't get the purchase order for these capital cost items past 
the accountants.


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com


--
https://mail.python.org/mailman/listinfo/python-list

Re: Yet Another Switch-Case Syntax Proposal

On Mon, Apr 7, 2014 at 3:49 AM, Marco S. elbar...@gmail.com wrote:
 switch day case in briefing_days:

 lunch_time = datetime.time(11, 30)
 meeting_time = datetime.time(12, 30)
 case not in briefing_days + festive_days:

 lunch_time = datetime.time(12)
 meeting_time = datetime.time(14)
 case in festive_days:

 go_to_work = False
 day_type = festive
 else:

 go_to_work = True
 day_type = ferial

 The if-else equivalent will be:

 if day in briefing_days:

 lunch_time = datetime.time(11, 30)
 meeting_time = datetime.time(12, 30)
 if day not in briefing_days + festive_days:

 lunch_time = datetime.time(12)
 meeting_time = datetime.time(14)
 if day in festive_days:

 go_to_work = False
 day_type = festive
 else:

 go_to_work = True
 day_type = ferial

Here's a simpler form of the proposal, which might cover what you
need. It's basically a short-hand if/elif tree.

case expression comp_op expression:
suite
case [comp_op] expression:
suite
...
else:
suite

This has a slight oddity of parsing (in that an expression can
normally have a comparison in it); if you really want to use the
result of a comparison inside a case block, you'd have to parenthesize
it. But it's easy enough to explain to a human.

case day in briefing_days:
lunch_time = datetime.time(11, 30)
meeting_time = datetime.time(12, 30)
case not in briefing_days + festive_days:
lunch_time = datetime.time(12)
meeting_time = datetime.time(14)
case in festive_days:
go_to_work = False
day_type = festive
else:
go_to_work = True
day_type = ferial

A case statement that opens with a comparison operator takes the value
from the previous case (without re-evaluating it); a case statement
that lacks a comparison altogether assumes == and does the above. In
either case (pardon the pun), the check will be done only if the
preceding case was false. An 'else' clause is effectively equivalent
to a 'case' that's always true.

Adds only one keyword to the language (switch is gone), and adds an
edge case to parsing that's unlikely to come up in non-contrived code.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

On Mon, Apr 7, 2014 at 4:09 AM, Mark Lawrence breamore...@yahoo.co.uk wrote:
 On 06/04/2014 18:27, Chris Angelico wrote:

 (plus, why on earth can't they afford a few more forks in the
 interests of hygiene??!?).


 They couldn't get the purchase order for these capital cost items past the
 accountants.

That would explain why they spend so much time thinking. Can't afford
pencils and paper to write their thoughts down, so they sit in the sun
and one by one they collect their thoughts and think them over.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

On Sunday, April 6, 2014 11:24:15 PM UTC+5:30, Chris Angelico wrote:
 On Mon, Apr 7, 2014 at 3:31 AM, Rustom Mody  wrote:
  However consider that some of the things that people did around 40 years
  ago and do today
  - use FORTRAN for numerical/simulation work --  now use scipy/sage etc
  - NLP with Lisp/Prolog -- look at Nltk
  - ??? with Data Analysis in Pandas
  - Scheme (basis for programming pedagogy, semantics research) - Python
  you can add/multiply ad libitum
  Yeah you covered this in your (2) as ...just a tool...
  Ask some recent PhD about what is for you just an almost irrelevant
  tool and you are very likely to find that that choice may well have
  been the difference between completing the research and giving up.
  I think python wins because it (usually) lets people do their thing


 [1] As the old joke goes: The physics department needs a whole lot of
 expensive equipment, but the math department needs only pencils,
 paper, and wastepaper baskets. And the philosophy department goes even
 further: they don't need wastepaper baskets.

HaHa! Very funny and unpleasantly accurate!


 Allow me to put it another way. Mathematicians use the language of
 algebra to describe their research; they don't, by and large, use a
 programming language. They use pencils and paper [1] as tools to get
 their work done, and may well have strong opinions on which pencil
 brand is the best, but the point of the pencil (pun intended) is to
 enable something else. It's supposed to get out of the way and let
 them do their thing. Python is highly practical because it gets out of
 the way. It's not the way that you develop programming language
 theory, though.

Right.
Whats wrong is (the implication -- maybe its not there??) that
CS begins and ends with develop programming language theory

 I might start out designing a language with the express purpose of
 implementing everything as an expression. The whole program consists
 of one long expression, with perhaps the semicolon being a sequence
 point that evaluates its left side, then evaluates its right side, and
 returns the latter (like the C comma operator). I could then go
 through a whole lot of lovely mental exploration as to what the
 benefits and costs of that system are, all the while writing nothing
 more than design documents and example code. At some point, if I'm
 happy with it, I'll write a reference implementation, and maybe then
 I'll use Python for the job. But that's not using Python to explore a
 language concept; that's using Python to rapidly prototype the
 executable code that I need in order to show my new language at work.
 All the work of developing the language is done in the design stage,
 with nothing at all even needing a computer (although I *guarantee*
 you that if I were to start something like that, I'd find part way
 through that I've made some fundamental mistakes early on - and a
 computer is better for editing text than anything on paper). I could
 just as easily write my reference implementation using yacc/bison and
 C, and it wouldn't be materially different.

Again I dispute the 'just'.

Its a right example for the wrong reason: State of art of writing
language implementations in python is no-better-probably-worse than
the venerable yacc ecosystem.
Choose an example where the difference between poor and good tool is more
palpable and the 'just' will no longer be upholdable as just.

Is the diff between cvs/svn and git just one vcs or another?

 Using Python at the design stage would be what Steven's talking about
 - actually using it to build the theory of programming. I have about
 as much experience in the area as he has, so we can't speak to the
 lack of examples, but that's the sort of example it would take.

!Parse Error! What are you saying -- I don get :-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

On Mon, Apr 7, 2014 at 4:13 AM, Rustom Mody rustompm...@gmail.com wrote:
 Is the diff between cvs/svn and git just one vcs or another?

The theory of version control, or source control, or whatever you want
to call it, can be found in some of the docs for those systems (git
goes into some depth about the Directed Acyclic Graph that underpins
everything), but that theory isn't what makes git or cvs/svn useful.

The theory behind my MUD client Gypsum is that it should be built
the way a server is, including that it should not need to be restarted
even when there's new code to be loaded in; but that's not what makes
Gypsum useful.

The theory behind an ergonomic keyboard is that it should hurt your
hands less than a classic keyboard does, but that's not what makes it
useful. Actually, in that instance, it might be what makes it
useless...

 Using Python at the design stage would be what Steven's talking about
 - actually using it to build the theory of programming. I have about
 as much experience in the area as he has, so we can't speak to the
 lack of examples, but that's the sort of example it would take.

 !Parse Error! What are you saying -- I don get :-)

What I'm saying is that I - and, if my reading is correct, similarly
with Steven - am looking for is a prominent example of someone using
Python as the very basis for a discussion on the future of computer
science *as a field*. So, not here's what can be done with Python,
and not here's something about hydraulics, with some Python code
showing how my theory adds up. If you're developing a cryptography
algorithm, it might well be convenient to support it with Python code
(although I mostly see reference implementations in C), but that's
still using Python as a tool, rather than as a language for
fundamental development of comp sci theories.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Yet Another Switch-Case Syntax Proposal

2014-04-06 Thread Michael Torrie

On 04/06/2014 12:07 PM, Chris Angelico wrote:
 This has a slight oddity of parsing (in that an expression can
 normally have a comparison in it); if you really want to use the
 result of a comparison inside a case block, you'd have to parenthesize
 it. But it's easy enough to explain to a human.

This syntax is almost identical to the if/elif/else syntax, though, no?

 
 case day in briefing_days:
 lunch_time = datetime.time(11, 30)
 meeting_time = datetime.time(12, 30)
 case not in briefing_days + festive_days:
 lunch_time = datetime.time(12)
 meeting_time = datetime.time(14)
 case in festive_days:
 go_to_work = False
 day_type = festive
 else:
 go_to_work = True
 day_type = ferial
 
 A case statement that opens with a comparison operator takes the value
 from the previous case (without re-evaluating it); a case statement
 that lacks a comparison altogether assumes == and does the above. In
 either case (pardon the pun), the check will be done only if the
 preceding case was false. An 'else' clause is effectively equivalent
 to a 'case' that's always true.
 
 Adds only one keyword to the language (switch is gone), and adds an
 edge case to parsing that's unlikely to come up in non-contrived code.
 
 ChrisA
 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

Steven D'Aprano steve+comp.lang.pyt...@pearwood.info:

 On Sun, 06 Apr 2014 12:05:16 +0300, Marko Rauhamaa wrote:
 Python, BTW, is perfectly suitable for computer science. 

 I don't think it is. Python is not a pure functional language, so it's 
 very difficult to prove anything about the code apart from running it. 

Many classic CS ideas are expressed in terms of an Algol-like language.
Nothing would prevent you from framing those ideas in a Python-like
(pseudo)language. The question is mostly whether you prefer begin/end,
braces or indentation.

  * combinatory birds in forests

 I don't believe that came from academia. If I've understood correctly, 
 that was from a non-academic book on applying the lambda calculus to 
 solve practical applications.

It is academic because the author, Raymond Smullyan, was a professor of
philosophy and, more importantly, my professor selected that as a
textbook for us graduate students.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

Chris Angelico ros...@gmail.com:

  * unfaithful husbands on an island ruled by female logicians

 I don't know that one.

 Me neither, although I can see elements of classic logic analysis
 elements. Islands ruled by logicians, people who always tell the truth
 / always tell exact falsehoods, etc, etc. I don't know of any that
 involve unfaithful husbands, but it wouldn't surprise me. Would like
 to hear it though.

Here's how I remember it:

  There was a tiny matriarchal island ruled by a queen. The women were
  capable logicians and that was common knowledge. The idyllic island had
  a problem, though: faithless husbands. The queen decided to solve the
  problem and summoned all women to the market square. She said:

We need to solve the problem of unfaithful husbands once and for
all. Now, we all know which men are cheating on their wives except
our own. I hereby ban you from talking about this matter with each
other ever again. However, if one day you should come to know your
husband has been unfaithful, I am ordering you to show no mercy but
shoot him to death the following night while he is asleep.

  The women left and went back to their business. The night after 40
  days, shots were heard throughout the island.

  How many husbands were unfaithful? How did they find out?

It was a variation of numerous similar puzzles and was the topic of a
dissertation on knowledge logic, IIRC.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Yet Another Switch-Case Syntax Proposal

2014-04-06 Thread Ian Kelly

On Sun, Apr 6, 2014 at 11:49 AM, Lucas Malor 3kywjyd...@snkmail.com wrote:
 On 3 April 2014 20:12, Ian Kelly ian.g.kelly-at-gmail.com
 |python-list@python.org| dnl5yyr...@sneakemail.com wrote:
 Use this instead [of continue]:


 switch day case in (Mon, Tue, Wed, Thu, Fri):
 go_to_work = True
 day_type = ferial
 if day in (Tue, Thu):
 lunch_time = datetime.time(11, 30)
 meeting_time = datetime.time(12, 30)
 else:
 lunch_time = datetime.time(12)
 meeting_time = datetime.time(14)
 case in (Sat, Sun):
 go_to_work = False
 day_type = festive

 You get an extra level of indentation this way, but it reads less like
 spaghetti code.


 Well, if you have to add an if-else to your switch-case, it means
 switch-case syntax is not so useful.

I agree; the above is better suited to be an if.

 An alternative is to imitate elif, so you'll have elcase. This way we don't
 need continue. Since I do not like elcasein, the best solution is to do as
 suggested by many of you, so case in instead of casein.

So if I'm understanding your intention correctly, case means to
check this case regardless of whether any preceding case was matched,
and elcase means to check this case only if the most recent case
and its dependent elcases preceding this one were not matched.

switch Mon case in (Tue, Thu):
print(1)
elcase in (Mon, Wed, Fri):
print(2)
case in (Sat, Sun):
print(3)
elcase in (Mon, Tue, Wed, Thu, Fri):
print(4)

will print 2 and 4, correct?

 But if you can write case in, why you can't write case comp_operator in
 general? With this syntax you can do also case is not, case  etc...

At this point I see no advantage in adding this syntax.  It is so
similar to an if-elif chain that surely any optimization that could be
applied would be equally possible if the if-elif syntax, so why not
just use that?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)


On 06/04/2014 21:10, Marko Rauhamaa wrote:


Many classic CS ideas are expressed in terms of an Algol-like language.
Nothing would prevent you from framing those ideas in a Python-like
(pseudo)language. The question is mostly whether you prefer begin/end,
braces or indentation.



Of course whilst all this work in the fields of languages, algorithms 
and such like has been going on, in parallel engineers have been working 
on the hardware side of things.  My understanding is that some abacuses 
now have as many as ten strings on them.  Although this scale was at 
first difficult for the users to grasp, the designers came up with the 
fantastic idea of using different coloured beads on different strings to 
simplify the user experience.


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com


--
https://mail.python.org/mailman/listinfo/python-list

Re: Yet Another Switch-Case Syntax Proposal

On Mon, Apr 7, 2014 at 6:13 AM, Michael Torrie torr...@gmail.com wrote:
 On 04/06/2014 12:07 PM, Chris Angelico wrote:
 This has a slight oddity of parsing (in that an expression can
 normally have a comparison in it); if you really want to use the
 result of a comparison inside a case block, you'd have to parenthesize
 it. But it's easy enough to explain to a human.

 This syntax is almost identical to the if/elif/else syntax, though, no?

Like I said, it's a short-hand for an if/elif tree, nothing more. Most
of the proposals have effectively been that anyway. There are
differences, though; the case target gets evaluated only once, for
instance. I'm not pushing strongly for its addition to the language.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

[ANN] ClamAV for Python 0.2!

2014-04-06 Thread Ryan Gonzalez

Announcing ClamAV for Python 0.2! ClamAV for Python is a set of pure-Python
bindings for libclamav. This version adds basic support for callbacks and
makes it work under Python 3. Check it out of
PyPIhttps://pypi.python.org/pypi/clamavand
GitHub https://github.com/kirbyfan64/clamav-python. Report bugs to the Issue
tracker https://github.com/kirbyfan64/clamav-python/issues.

-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

2014-04-06 Thread Steven D'Aprano

On Sun, 06 Apr 2014 23:10:47 +0300, Marko Rauhamaa wrote:

 Steven D'Aprano steve+comp.lang.pyt...@pearwood.info:
 
 On Sun, 06 Apr 2014 12:05:16 +0300, Marko Rauhamaa wrote:
 Python, BTW, is perfectly suitable for computer science.

 I don't think it is. Python is not a pure functional language, so it's
 very difficult to prove anything about the code apart from running it.
 
 Many classic CS ideas are expressed in terms of an Algol-like language.
 Nothing would prevent you from framing those ideas in a Python-like
 (pseudo)language. The question is mostly whether you prefer begin/end,
 braces or indentation.

Okay, I made an error in stating that it's because Python is not a pure 
functional language. It's because Python is so dynamic that it is very 
difficult to prove anything about the code apart from running it. Take 
this code-snippet of Python:

n = len([1, 2, 3])

What can we say about it? Almost nothing!

All we know is that the name len will be looked up, it may or may not 
find something, that thing may or may not be callable, calling it with a 
list may or may not succeed, and it may or may not return 3 when given 
that specific list as input. From the perspective of wanting to prove 
things about the code, there's not a lot of certainty there.

If we replace Python with a Python-like language which is closer to the 
traditional Algol mode of built-in functions being keywords (and hence 
unable to be shadowed or deleted) then we can reason about the behaviour 
more successfully. Alas, a Python-like language is not Python, and our 
discussion is about whether or not *Python* is suitable for this use.


  * combinatory birds in forests

 I don't believe that came from academia. If I've understood correctly,
 that was from a non-academic book on applying the lambda calculus to
 solve practical applications.
 
 It is academic because the author, Raymond Smullyan, was a professor of
 philosophy and, more importantly, my professor selected that as a
 textbook for us graduate students.

Ah. Well they do that, don't they? I've always consider the ability of 
professors to select their own book as text to be a classic case of 
conflict of interest. They're supposed to pick the best book, not 
necessarily the one that earns them money.



-- 
Steven D'Aprano
http://import-that.dreamwidth.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

2014-04-06 Thread Terry Reedy


On 4/6/2014 7:48 PM, Steven D'Aprano wrote:

On Sun, 06 Apr 2014 23:10:47 +0300, Marko Rauhamaa wrote:


Steven D'Aprano steve+comp.lang.pyt...@pearwood.info:


On Sun, 06 Apr 2014 12:05:16 +0300, Marko Rauhamaa wrote:

Python, BTW, is perfectly suitable for computer science.


I don't think it is. Python is not a pure functional language, so it's
very difficult to prove anything about the code apart from running it.


Many classic CS ideas are expressed in terms of an Algol-like language.
Nothing would prevent you from framing those ideas in a Python-like
(pseudo)language. The question is mostly whether you prefer begin/end,
braces or indentation.


Okay, I made an error in stating that it's because Python is not a pure
functional language. It's because Python is so dynamic that it is very
difficult to prove anything about the code apart from running it. Take
this code-snippet of Python:

n = len([1, 2, 3])

What can we say about it? Almost nothing!


One merely needs to stipulate that builtin names have not been rebound 
to give the answer: n is bound to 3. In the absence of code or text 
specifying otherwise, that is the reasonable default assumption and the 
one that most makes when reading code.


Restricting the usage of Python's flexibility does not make it another 
language. It makes it the actual language that the vast majority of 
programs are written in and that people assume when reading code.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

On Monday, April 7, 2014 6:15:47 AM UTC+5:30, Terry Reedy wrote:
 On 4/6/2014 7:48 PM, Steven D'Aprano wrote:
  On Sun, 06 Apr 2014 23:10:47 +0300, Marko Rauhamaa wrote:
  Steven D'Aprano :
  On Sun, 06 Apr 2014 12:05:16 +0300, Marko Rauhamaa wrote:
  Python, BTW, is perfectly suitable for computer science.
  I don't think it is. Python is not a pure functional language, so it's
  very difficult to prove anything about the code apart from running it.
  Many classic CS ideas are expressed in terms of an Algol-like language.
  Nothing would prevent you from framing those ideas in a Python-like
  (pseudo)language. The question is mostly whether you prefer begin/end,
  braces or indentation.
  Okay, I made an error in stating that it's because Python is not a pure
  functional language. It's because Python is so dynamic that it is very
  difficult to prove anything about the code apart from running it. Take
  this code-snippet of Python:
  n = len([1, 2, 3])
  What can we say about it? Almost nothing!

 One merely needs to stipulate that builtin names have not been rebound 
 to give the answer: n is bound to 3. In the absence of code or text 
 specifying otherwise, that is the reasonable default assumption and the 
 one that most makes when reading code.

 Restricting the usage of Python's flexibility does not make it another 
 language. It makes it the actual language that the vast majority of 
 programs are written in and that people assume when reading code.

Well what Steven is saying (I think!) amounts to pointing out a gap:
- the actual language that the vast majority of programs are written in 
- python as in the Cpython (say) implementation

To close this gap requires trying to do what Brett Canon tried and more
generally PyPy tries.

Some small rarely used features which humans can invoke with with a
wave-fo-hands when reasoning about programs, end up being a
show-stopper in an implementation.

Every language has such: Fortran remains better for scientific computing
than C (leave aside C++) because among other things alias analysis for
Fortran arrays is more straightforward.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

On Monday, April 7, 2014 12:16:54 AM UTC+5:30, Chris Angelico wrote:

On Mon, Apr 7, 2014 at 4:13 AM, Rustom Mody wrote:
Using Python at the design stage would be what Steven's talking about
- actually using it to build the theory of programming. I have about
as much experience in the area as he has, so we can't speak to the
lack of examples, but that's the sort of example it would take.

!Parse Error! What are you saying -- I don get :-)

What I'm saying is that I - and, if my reading is correct, similarly
with Steven - am looking for is a prominent example of someone using
Python as the very basis for a discussion on the future of computer
science *as a field*. So, not here's what can be done with Python,
and not here's something about hydraulics, with some Python code
showing how my theory adds up. If you're developing a cryptography
algorithm, it might well be convenient to support it with Python code
(although I mostly see reference implementations in C), but that's
still using Python as a tool, rather than as a language for
fundamental development of comp sci theories.

Nice example

10 years ago Nicholas Carr wrote an article: Does IT matter?
http://hbr.org/2003/05/it-doesnt-matter/ar/1

| Twenty years ago, most executives looked down on computers as
| proletarian tools—glorified typewriters and calculators—best relegated
| to low level employees like secretaries, analysts, and technicians. It
| was the rare executive who would let his fingers touch a keyboard,
| much less incorporate information technology into his strategic
| thinking. Today, that has changed completely. Chief executives now
| routinely talk about the strategic value of information technology...
|
| Behind the change in thinking lies a simple assumption: that as IT’s
| potency and ubiquity have increased, so too has its strategic
| value. It’s a reasonable assumption, even an intuitive one. But it’s
| mistaken. What makes a resource truly strategic—what gives it the
| capacity to be the basis for a sustained competitive advantage—is not
| ubiquity but scarcity. You only gain an edge over rivals by having or
| doing something that they can’t have or do. By now, the core functions
| of IT—data storage, data processing, and data transport—have become
| available and affordable to all.1 Their very power and presence have
| begun to transform them from potentially strategic resources into
| commodity factors of production. They are becoming costs of doing
| business that must be paid by all but provide distinction to none.

Now replace IT by CS.

CS matters because it has stopped being visible -- entered the woodword.
This is the underlying principle of python replacing scheme for programming
at MIT. Its not that python is a better language. Its rather that
doing the job and getting out of the way is more crucial today than 1980.
http://cemerick.com/2009/03/24/why-mit-now-uses-python-instead-of-scheme-for-its-undergraduate-cs-program/

So cryptographic algos need (typically)
1. An algorithmic language
2. Fast implementations
Python only provides 1, C provides both.So C is more useful (ignoring the
marginal effects of inertia)

Is the diff between cvs/svn and git just one vcs or another?

The theory of version control, or source control, or whatever you want
to call it, can be found in some of the docs for those systems (git
goes into some depth about the Directed Acyclic Graph that underpins
everything), but that theory isn't what makes git or cvs/svn useful.

The theory behind my MUD client Gypsum is that it should be built
the way a server is, including that it should not need to be restarted
even when there's new code to be loaded in; but that's not what makes
Gypsum useful.

The theory behind an ergonomic keyboard is that it should hurt your
hands less than a classic keyboard does, but that's not what makes it
useful. Actually, in that instance, it might be what makes it
useless...

These examples are very different:
1. MUD I dont know
2. Ergonomic keyboard is a good example.
For a ergonomic keyboard to be useful it has to satisfy the precondition
Not more than a δ neighborhood away from QWERTY
3. Git: We differ on whats the underlying theory. For me crucial is
a. Peer-to-peer replacing client-server -- this gives the D in DVCS
b. Branching as central to software (more generally any material) development
--
https://mail.python.org/mailman/listinfo/python-list

threading

2014-04-06 Thread Onder Hazaroglu

Hello,
I've been using threading library to run some experiments parallel. There is
no message passing between my threads but still it messes up somehow. The
results are different than running it separated. Basically I experiment with
three threads working on three different files but the results are different
than running three of them sequentially. Is there a way to make sure that
there is no memory sharing between threads?

-- 
Best Regards,

Onder HAZAROGLU | Graduate Student | Ph.D. Candidate
University of Arkansas at Little Rock | Computer Science Department
EIT Bldg. | (501) 615-3851 | oxhazaro...@ualr.edu, onderhazaro...@gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading

2014-04-06 Thread Ben Finney

Onder Hazaroglu oxhazaro...@ualr.edu writes:

 I've been using threading library to run some experiments parallel.

Threading is very difficult to get right, much more so than the usual
separation into distinct processes. How did you decide against the
normal means of parallel execution and instead choose threading?

 There is no message passing between my threads but still it messes up
 somehow. The results are different than running it separated.

Yes, this is a common result of threading; problems are much harder to
reproduce and much more entangled with apparently unrelated factors.
This is one of the main reasons to avoid threading if at all feasible.

 Is there a way to make sure that there is no memory sharing between
 threads?

The *whole point* of threading (AFAIK) is to share memory and other
process-distinct resources. If you're looking to avoid that, don't use
threading! Instead, use separate processes for parallel execution.

Parallel processing is achieved much more reliably and deterministically
with separate processes. Python even makes this much easier than most
languages, with the ‘multiprocessing’ module in the standard library
URL:https://docs.python.org/3/library/multiprocessing.html.

I would recommend you make easier-to-understand and easier-to-debug code
with that module, and only consider threading if you determine it's
needed.

-- 
 \“Stop — Drive sideways.” —detour sign, Kyushu, Japan |
  `\   |
_o__)  |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading

On Monday, April 7, 2014 8:24:37 AM UTC+5:30, Onder Hazaroglu wrote:
 Hello,
 I've been using threading library to run some experiments parallel. There is
 no message passing between my threads but still it messes up somehow. The
 results are different than running it separated. Basically I experiment with
 three threads working on three different files but the results are different
 than running three of them sequentially. Is there a way to make sure that
 there is no memory sharing between threads?

Python -- like most languages -- does not provide the option:
thread-efficiency plus process-isolation.

Erlang makes this its centerpiece. So look at it if thats what you are
after.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python streaming media server

2014-04-06 Thread Wesley

在 2014年4月6日星期日UTC+8下午8时52分37秒，Sturla Molden写道：
 Wesley nisp...@gmail.com wrote:
 
 
 
  Not open source, but there is a famous closed-source one called YouTube.
 
  
 
  Are you kidding?
 
  I know youtube, but do you think we can use it setup our own streaming 
  media server?
 
 
 
 Obviously not. 
 
 
 
 Before YouTube was bought by Google, it was common knowledge that it ran on
 
 Stackless Python. So a streaming media server on Python is absolutely
 
 possible. But no, I don't know of one you can set up and use on your own.
 
 
 
 You can make a highly scalable server with PyZMQ and Tornado or Twisted.
 
 NumPy is great for storing binary data like media streams. HDF5 (PyTables
 
 or h5py) might be a better database than some SQL server, as it is capable
 
 of highly scalable parallel binary i/o.  
 
 
 
 Sturla

Thanks, Sturla.
Umm,I think we can setup one using the technique skills you mentioned above:-)
But that will need a lot work to do I think.

I am looking at an opensource one named Flumotion.

Wesley
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading

2014-04-06 Thread Roy Smith

In article mailman.8969.1396839923.18130.python-l...@python.org,
 Ben Finney ben+pyt...@benfinney.id.au wrote:

 The *whole point* of threading (AFAIK) is to share memory and other
 process-distinct resources.

There is (or at least, was) another reason.  Creating a new process used 
to be far more expensive than creating a new thread.  In modern  Unix 
kernels, however, the cost difference has become much less, so this is 
no longer a major issue.

I agree wholeheartedly with Ben when he says:

 Parallel processing is achieved much more reliably and deterministically
 with separate processes.

Threading makes it incredibly difficult to reason about program 
execution.  It's not just that things happen asynchronously, the control 
flow changes happen at arbitrary times.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading

On Mon, Apr 7, 2014 at 1:48 PM, Roy Smith r...@panix.com wrote:
 There is (or at least, was) another reason.  Creating a new process used
 to be far more expensive than creating a new thread.  In modern  Unix
 kernels, however, the cost difference has become much less, so this is
 no longer a major issue.

Unix maybe, but what about Windows? Is it efficient to create
processes under Windows?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

Steven D'Aprano steve+comp.lang.pyt...@pearwood.info:

 On Sun, 06 Apr 2014 23:10:47 +0300, Marko Rauhamaa wrote:
 It is academic because the author, Raymond Smullyan, was a professor
 of philosophy and, more importantly, my professor selected that as a
 textbook for us graduate students.

 Ah. Well they do that, don't they? I've always consider the ability of
 professors to select their own book as text to be a classic case of
 conflict of interest. They're supposed to pick the best book, not
 necessarily the one that earns them money.

Note that my professor above was not Raymond Smullyan.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

2014-04-06 Thread Steven D'Aprano

On Sun, 06 Apr 2014 20:45:47 -0400, Terry Reedy wrote:

 On 4/6/2014 7:48 PM, Steven D'Aprano wrote:
 On Sun, 06 Apr 2014 23:10:47 +0300, Marko Rauhamaa wrote:

 Steven D'Aprano steve+comp.lang.pyt...@pearwood.info:

 On Sun, 06 Apr 2014 12:05:16 +0300, Marko Rauhamaa wrote:
 Python, BTW, is perfectly suitable for computer science.

 I don't think it is. Python is not a pure functional language, so
 it's very difficult to prove anything about the code apart from
 running it.

 Many classic CS ideas are expressed in terms of an Algol-like
 language. Nothing would prevent you from framing those ideas in a
 Python-like (pseudo)language. The question is mostly whether you
 prefer begin/end, braces or indentation.

 Okay, I made an error in stating that it's because Python is not a pure
 functional language. It's because Python is so dynamic that it is very
 difficult to prove anything about the code apart from running it. Take
 this code-snippet of Python:

 n = len([1, 2, 3])

 What can we say about it? Almost nothing!
 
 One merely needs to stipulate that builtin names have not been rebound
 to give the answer: n is bound to 3. 

But if I can do that, I can also stipulate that len() has been rebound to 
a function that ignores its argument and always returns the string 
Surprise!. In that case, n is bound to the string Surprise!. I can 
prove that this code snippet does almost *anything*, just be making some 
assumption about len.

The point is that one cannot derive much about the behaviour of Python 
code except by analysing the whole program, which is a very difficult 
problem, and often not even then. The only way to be sure what value is 
bound to len at the time that code snippet is executed is to actually run 
the code up to that code snippet and then look. In practical terms, 
things are not quite as bleak as I've made out: a little bit of runtime 
analysis goes a long way, as the success of PyPy, Numba, Cython and Psyco 
prove. That's why optimizers like PyPy generally produce code like this:

if some guard condition is true:
run fast optimized branch
else:
fall back on standard Python 

where the guard condition is generally checked at runtime, not at compile 
time.

But *in isolation*, you can't tell what len will be bound to unless you 
wait until runtime and look. A peephole optimizer that replaced a call 
like len([1,2,3]) with the constant 3 every time it sees it would be 
*wrong*.


 In the absence of code or text
 specifying otherwise, that is the reasonable default assumption and the
 one that most makes when reading code.

Well of course, but the requirements of an optimizer or correctness 
prover or similar is much higher than just this is a reasonable default 
assumption. 


 Restricting the usage of Python's flexibility does not make it another
 language. It makes it the actual language that the vast majority of
 programs are written in and that people assume when reading code.

That's incorrect. If len were a keyword, and couldn't be shadowed or 
replaced, it would be another language. It is not an accident that you 
can replace len in builtins, it is a deliberate feature of the language.



-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading

Ben Finney ben+pyt...@benfinney.id.au:

 The *whole point* of threading (AFAIK) is to share memory and other
 process-distinct resources.

Another way to look at it is that threads were pushed as a magic bullet
to manage the complexities of network programming. They were fashionable
in Windows and Java. The idea was that the programmer could write linear
code that blocks on I/O and not be overwhelmed by the intricacies.

I don't think it worked out all that well.

Since then both Windows and Java have come up with their own I/O
multiplexing facilities. Now we see Python follow suit with asyncio.

Threads have their uses, but they are such tricky beasts that I would
see first if the problem couldn't be handled with multiplexing and
multiprocessing.

The main need for threads today comes from the database libraries,
which, AFAIK, generally don't provide a nonblocking API.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

Steven D'Aprano st...@pearwood.info:

 That's why optimizers like PyPy generally produce code like this:

 if some guard condition is true:
 run fast optimized branch
 else:
 fall back on standard Python 

There you go! You are using Python-esque syntax to communicate a CS
idea.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading

2014-04-06 Thread Ben Finney

Chris Angelico ros...@gmail.com writes:

 On Mon, Apr 7, 2014 at 1:48 PM, Roy Smith r...@panix.com wrote:
  There is (or at least, was) another reason. Creating a new process
  used to be far more expensive than creating a new thread. In modern
  Unix kernels, however, the cost difference has become much less, so
  this is no longer a major issue.

 Unix maybe, but what about Windows? Is it efficient to create
 processes under Windows?

Another reason to avoid Microsoft's operating systems as a programming
target, IMO.

-- 
 \“We cannot solve our problems with the same thinking we used |
  `\   when we created them.” —Albert Einstein |
_o__)  |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading

2014-04-06 Thread Paul Rubin

Marko Rauhamaa ma...@pacujo.net writes:
 Since then both Windows and Java have come up with their own I/O
 multiplexing facilities. Now we see Python follow suit with asyncio.

That all happened because threads in those systems are rather expensive.
GHC and Erlang have fast lightweight threads/processes and programming
with them is much more civilized than using async schemes.  Even a low
level language like Forth reached something similar.

I keep hearing about all the perils of threading bugs and it just hasn't
happened to me in Python as far as I know.  The main trick is to not
share any mutable data between threads.  Instead have them communicate
by message passing through Queues.  If you've got a lot of tasks in the
system then it helps to have a bit of abstraction to keep the queues
organized and make the other tasks addressible by name, but it's all
pretty straightforward.  You do take an efficiency hit, but if that's a
big concern you sort of have to look past Python.

Lately I'm messing with Go and it's sort of the same idea.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading

Paul Rubin no.email@nospam.invalid:

 I keep hearing about all the perils of threading bugs and it just
 hasn't happened to me in Python as far as I know.

Good for you. I'm saying the first step to thread-safe code is to have a
healthy fear of the perils.

 The main trick is to not share any mutable data between threads.
 Instead have them communicate by message passing through Queues.

That certainly is a good way and is akin to multiprocessing. You still
need to make sure you don't flood the queues or cause deadlocks by
limiting queue sizes.

It is still easy to accidentally pass references to mutable objects
between threads (and most people don't even try to avoid it).
Multiprocessing naturally enforces the principle.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue21130] equivalent functools.partial instances should compare equal

2014-04-06 Thread Raymond Hettinger


Raymond Hettinger added the comment:

 I guess a partial is a binding, not a function.

Viewed in that light, it makes sense to implement same logic as used for bound 
methods in method_richcompare.

--
keywords: +easy
stage: test needed - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21130
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21111] PyLong_AsUnsignedLongAndOverflow does not exist

2014-04-06 Thread Hristo Venev


Hristo Venev added the comment:

I will not add PyLong_AsUnsigned*AndOverflow in my code because I don't want my 
code to depend on the exact implementation of PyLong.

Are you seriously calling a 50-line function feature?

Anyway... I propose splitting the patch in two parts:
   - cleanup: the changes to Objects/longobject.c
   - feature: the changes to Include/longobject.h

cleanup can be applied to 3.4.1 because it adds no new features and helps 
maintainability. Backwards compatibility will not be broken.
feature can then be added in 3.5. Backwards compatibility should not be 
broken.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21111] PyLong_AsUnsignedLongAndOverflow does not exist

2014-04-06 Thread Mark Dickinson


Mark Dickinson added the comment:

Thanks for the patch!  I left a couple of comments on Rietveld.  The patch will 
also need docs and tests before it's ready to go in.

There's a behaviour change in the patch which needs looking into: before the 
patch, PyLong_AsUnsignedLong will not call the target object's __int__ method 
(so it should raise a TypeError for a float or Decimal instance, for example).  
It looks to me as though the patch changes that;  was that change intentional?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21144] ensurepip AssertionError: Multiple .dist-info directories

2014-04-06 Thread Samuel John


Changes by Samuel John pyt...@samueljohn.de:


--
nosy: +samueljohn

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21144
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20383] Add a keyword-only spec argument to types.ModuleType

2014-04-06 Thread Nick Coghlan


Nick Coghlan added the comment:

No, the attribute level arguments won't go away - __name__ deliberately differs 
from __spec__.name in some cases (notably in __main__), __path__ may be 
manipulated after the module is loaded, and __name and __file__ are both used 
too heavily within module code for it to be worth the hassle of deprecating 
them in favour of something else.

I think Brett's push to simplify things as much as possible is good though - 
that's the main brake on creeping API complexity in the overall import system 
as we try to make the internals easier to comprehend and manipulate.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20383
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21145] Add the @cached_property decorator

2014-04-06 Thread Omer Katz


Omer Katz added the comment:

The default implementation should simple:
Once the property is accessed for the first time calculate the result and
never calculate again.
It's what both Django  pip uses.
You can add an optional TTL.
There aren't any other features I can come up with that are common enough.
If needed, one can inherit from cached_property and do whatever is needed.
Eric, Why don't you think a C implementation is needed? It's a simple
operation for sure but it is meant to increase runtime efficiency.

2014-04-05 1:11 GMT+04:00 Éric Araujo rep...@bugs.python.org:


 Éric Araujo added the comment:

 It could make sense to add clean, working recipes to e.g. the functools
 documentation.  The cached_property in the wiki uses a TTL, other like
 Pyramid's reify decorator make properties that ensure the fget function is
 called only once per instance, and there may be subtly different variants
 out there.  I don't know if there's a universally useful variant that
 should be added to the sdlib right now.  (I don't think a C implementation
 is needed.)

 On a related note, the Python docs about desciptors may be missing
 entry-level explanations, as described here:
 http://me.veekun.com/blog/2012/05/23/python-faq-descriptors/

 --
 nosy: +eric.araujo, rhettinger

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue21145
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21145
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21145] Add the @cached_property decorator

2014-04-06 Thread Omer Katz


Omer Katz added the comment:

I just checked and werkzeug uses the same implementation as Django  pip.

2014-04-06 14:31 GMT+04:00 Omer Katz rep...@bugs.python.org:


 Omer Katz added the comment:

 The default implementation should simple:
 Once the property is accessed for the first time calculate the result and
 never calculate again.
 It's what both Django  pip uses.
 You can add an optional TTL.
 There aren't any other features I can come up with that are common enough.
 If needed, one can inherit from cached_property and do whatever is needed.
 Eric, Why don't you think a C implementation is needed? It's a simple
 operation for sure but it is meant to increase runtime efficiency.

 2014-04-05 1:11 GMT+04:00 Éric Araujo rep...@bugs.python.org:

 
  Éric Araujo added the comment:
 
  It could make sense to add clean, working recipes to e.g. the functools
  documentation.  The cached_property in the wiki uses a TTL, other like
  Pyramid's reify decorator make properties that ensure the fget function
 is
  called only once per instance, and there may be subtly different variants
  out there.  I don't know if there's a universally useful variant that
  should be added to the sdlib right now.  (I don't think a C
 implementation
  is needed.)
 
  On a related note, the Python docs about desciptors may be missing
  entry-level explanations, as described here:
  http://me.veekun.com/blog/2012/05/23/python-faq-descriptors/
 
  --
  nosy: +eric.araujo, rhettinger
 
  ___
  Python tracker rep...@bugs.python.org
  http://bugs.python.org/issue21145
  ___
 

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue21145
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21145
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20397] distutils --record option does not validate existance of byte-compiled files

2014-04-06 Thread koobs


Changes by koobs koobs.free...@gmail.com:


--
nosy: +koobs

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20397
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21109] tarfile: Traversal attack vulnerability

2014-04-06 Thread Lars Gustäbel


Lars Gustäbel added the comment:

In the past, our answer to these kinds of bug reports has always been that you 
must not extract an archive from an untrusted source without making sure that 
it has no malicious contents. And that tarfile conforms to the posix 
specifications with respect to extraction of files and pathname resolution.  
That's why we put this prominent warning in the documentation, and I think its 
advice still holds.

I don't think that this issue should be marked as a release blocker, because 
the way tarfile currently works was a conscious decision, not an accident.  
tarfile does what it is designed to do: it processes a sequence of instructions 
to store a number of files in the filesystem. So the attack that is described 
by Daniel Garcia exploits neither a bug in tarfile nor a loophole in the tar 
archive format. A necessary condition for this attack to work is that the 
attacker has to trick the user into extracting the malicious archive first. 
After that, tarfile interprets the contained instructions word-for-word but 
still only within the boundaries defined by the user's privileges.

I think it is obvious that it is potentially dangerous to extract tar archives 
we didn't create ourselves, because we actually give another person direct 
access to our filesystem. tarfile could mitigate some of the adverse effects, 
but this will not change the fact that it remains unsafe to use tarfile to a 
certain degree unless you use it with your own data or take reasonable 
precautions.

Anyway, if we come to the conclusion that we want to eliminate this kind of 
attack, we must be aware that there is a lot more to do than that. tarfile as 
it is today is vulnerable to all known attacks against tar programs, and maybe 
even a few more that rely on its specific implementation.


1. Path traversal:

The archive contains files names e.g. /etc/passwd or ../etc/passwd.

2. Symlink file attack:

foo links to /etc/passwd.
Another member named foo follows, its data overwrites the target file's 
data.

3. Symlink directory attack:

foo links to /etc.
The following member foo/passwd overwrites /etc/passwd.

4. Hardlink attack:

Hardlink member foo links to /etc/passwd.
tarfile creates the hardlink to /etc/passwd because it cannot find it 
inside the archive and falls back to the one in the filesystem.
Another file named foo follows, its data overwrites /etc/passwd's data.

5. Permission manipulation:

The archive contains an executable that is placed somewhere in PATH with 
its setuid flag set, so that an unprivileged user is able to gain root 
privileges.

6. Device file attacks:

The archive contains a device node foo with the same major and minor 
numbers as an attached device.
Another member named foo follows, its data is written to the device.

7. Huge zero file attacks:

Bzip2 and lzma allow it to store huge blobs of repetetive data in tiny 
archives. When unpacked this data may fill up an entire filesystem.

8. Excessive memory usage:

tarfile saves one TarInfo object per member it finds in an archive. If the 
archive contains several millions of members, this may fill up the memory.

9. Saving a huge sparse file:

tarfile is unable to detect holes in sparse files and thus cannot store 
them efficiently. Archiving a huge sparse file can take very long and may lead 
to a very big archive that fills up the filesystem.


Additionally, there are more issues mentioned in the GNU tar manual:

  https://www.gnu.org/software/tar/manual/html_node/Security.html


In conclusion, I like to emphasize that tarfile is a library, it is no 
replacement for GNU tar. And as a library it has a different focus, it is 
merely a building block for an application, and has to be used with a little 
bit of responsibility. And even if we start to implement all possible checks, 
I'm afraid we never can do without a warning in the documentation that reminds 
everyone to keep an eye on what they're doing.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21109
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15955] gzip, bz2, lzma: add option to limit output size

2014-04-06 Thread Nadeem Vawda


Nadeem Vawda added the comment:

I've posted a review at http://bugs.python.org/review/15955/. (For some reason, 
it looks like Rietveld didn't send out email notifications. But maybe it never 
sends a notification to the sender? Hmm.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21109] tarfile: Traversal attack vulnerability

2014-04-06 Thread Larry Hastings


Larry Hastings added the comment:

Thank you Lars for your thorough reply.

While I agree that this isn't a release blocker, as it was clearly designed to 
behave this way... it seems to me that it wouldn't take much to make the 
tarfile module a lot safer.  Specifically:

  * Don't allow creating files whose absolute path is not under the
destination.
  * Don't allow creating links (hard or soft) which link to a path
outside of the destination.
  * Don't create device nodes.

This would fix your listed attacks 1-6.  The remaining attacks you cite are 
denial-of-service attacks; while they're undesirable, they shouldn't compromise 
the security of the machine.  (I suppose we could even address those, adding 
reasonable quotas for disk space and number of files.)

I doubt that would make tarfile secure.  But maybe practicality beats purity?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21109
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21111] PyLong_AsUnsignedLongAndOverflow does not exist

2014-04-06 Thread Hristo Venev


Hristo Venev added the comment:

I did not intend to make it so that PyLong_AsUnsignedLong* to call __int__ but 
it looks like a good idea because PyLong_AsLong* does that and the patch is 
exactly about that - removing differences between converting to signed and 
unsigned.

Should I upload the patch with docs and with the warning fixed? Will it be 
applied in 3.4.1? It will be a lot easier to check the value of an int than to 
create a new PyLong = 2^64 and compare the number with that.

P.S.: is there a very fast way to check if a PyLong fits in unsigned long long 
with the limited API?
P.P.S.: Just a random idea: would it be a to rewrite PyLong to use GMP instead 
as a PyVarObject of mp_limb_t's?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21111] PyLong_AsUnsignedLongAndOverflow does not exist

2014-04-06 Thread Mark Dickinson


Mark Dickinson added the comment:

 it looks like a good idea

It's not a good idea, though: you risk breaking existing third-party extension 
modules in surprising ways, which isn't a friendly thing to do.  In any case, 
if anything the PyLong methods should be moving *away* from use of the nb_int 
slot.  This has been discussed multiple times previously; one reason that we 
haven't made that change yet is the problem of breaking backwards compatibility.

 Should I upload the patch with docs and with the warning fixed?

Sure, that would be great if you have the time.  Tests would be useful too, but 
one of us can fill those in (eventually).  It all depends how much of a hurry 
you're in. :-)

 Will it be applied in 3.4.1?

No: as already explained by others, it's a enhancement, not a bugfix, so it 
can't be applied until 3.5.

 P.S.: is there a very fast way to check if a PyLong fits in unsigned long 
 long with the limited API?

Depends on your definition of 'very fast'.  There should be no need to compare 
with a constant: just try the conversion with PyLong_AsUnsignedLong and see if 
you get an OverflowError back.  My guess is that that's not going to be much 
slower than a custom PyLong_AsUnsignedLongAndOverflow, but the only way you'll 
find out is by timing your particular use-case.

 P.P.S.: Just a random idea: would it be a to rewrite PyLong to use GMP 
 instead as a PyVarObject of mp_limb_t's?

I'll let Victor answer that one. :-)  In the mean time, see issue 1814.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21139] Idle: change default reformat width from 70 to 72

2014-04-06 Thread Saimadhav Heblikar


Changes by Saimadhav Heblikar saimadhavhebli...@gmail.com:


--
nosy: +sahutd

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21139
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15955] gzip, bz2, lzma: add option to limit output size

2014-04-06 Thread Serhiy Storchaka


Serhiy Storchaka added the comment:

Yes, Rietveld never sends a notification to the sender. I've received a 
notification.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21145] Add the @cached_property decorator

2014-04-06 Thread R. David Murray


Changes by R. David Murray rdmur...@bitdance.com:


--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21145
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20383] Add a keyword-only spec argument to types.ModuleType

2014-04-06 Thread Brett Cannon


Brett Cannon added the comment:

I can dream about getting rid of the attributes, but I doubt it would happen 
any time soon, if at all. But we do need to make it easier to set __spec__ on a 
new module than it currently is to help promote its use.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20383
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit

2014-04-06 Thread Richard Oudkerk


Richard Oudkerk added the comment:

I would guess that the problem is simply that LogisticRegression objects are 
not picklable.  Does the problem still occur if you do not use freeze?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21162
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit


Ivan K added the comment:

Sorry, I'm not sure I describe it correct. Freeze that means = goes fozen, so 
stop progress. It's do no do anything, but computations still load single core 
of my cpu for 100% untill I do not kill the python process.

But the same code work's fine if executed outside pool, or on the different 
platforms (like Mac for example), my friend run it successfully, but for me it 
not works. So it's look like pretty unpredictable issue with conflicting OS, 
CPU, and Python I'm afraid.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21162
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit

2014-04-06 Thread Richard Oudkerk


Richard Oudkerk added the comment:

Ah, I misunderstood: you meant that it freezes/hangs, not that you used a 
freeze tool.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21162
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit


Ivan K added the comment:

Yes, I'm not using any tool. Code just not working.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21162
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20998] fullmatch isn't matching correctly under re.IGNORECASE

2014-04-06 Thread Gareth Gouldstone


Gareth Gouldstone added the comment:

fullmatch() is not yet implemented on the regex scanner object SRE_Scanner 
(issue 21002). Is it possible to adapt this patch to fix this omission?

--
nosy: +Gareth.Gouldstone

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20998
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1191964] asynchronous Subprocess

2014-04-06 Thread Josiah Carlson


Josiah Carlson added the comment:

Should have uploaded this yesterday, but I got caught up with typical weekend 
activities. The docs probably need more, and I hear that select.select() is 
disliked, so that will probably need to be changed too.

--
Added file: http://bugs.python.org/file34743/subprocess.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1191964
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit

2014-04-06 Thread Richard Oudkerk


Richard Oudkerk added the comment:

Could you try pickling and unpickling the result of func():

import cPickle
data = cPickle.dumps(func([1,2,3]), -1)
print cPickle.loads(data)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21162
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit


Ivan K added the comment:

Sorry, could you specify what is 'func' ?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21162
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit


Ivan K added the comment:

Sorry, stupdi question. Forget that this is from my gist

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21162
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit