Re: Writing a chess-playing AI like Alphago in Python

[Steve D'Aprano]

On Sun, 24 Dec 2017 12:20 pm, Cai Gengyang wrote:

> How many lines of code in Python would it take to create a Go-playing AI
> like AlphaGo ? Estimates ?

Somewhere between 1 and 1 billion.

How about you start by telling us:

- do you mean AlphaGo or AlphaGo Zero?

- how many lines of code AlphaGo [Zero] has;

- in what language or languages;

- is Python allowed to call out to libraries written in other 
  languages, e.g. machine learning and neural net libraries,
  or databases, or does it have to implement *everything*
  from scratch?


The Michi Go engine uses about 550 lines of Python:

https://github.com/pasky/michi

but I don't believe it does any machine learning.


See also:

https://github.com/rossumai/nochi

https://medium.com/rossum/building-our-own-version-of-alphago-zero-b918642bd2b5



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What is the meaning of @@?

[Steve D'Aprano]

On Sat, 23 Dec 2017 04:38 pm, Peng Yu wrote:

> Hi, I only can find the doc for @. What does @@ mean in python?

I don't think that @@ means anything yet.

There was a proposal to use @@ for matrix exponentiation in Numpy, as @ is
used for matrix multiplication, but that was left on hold to see whether it
is really needed or not.

Where did you find @@ in Python?


(By the way, @ for matrix multiplication only works in Python 3.5 or better.)

https://www.python.org/dev/peps/pep-0465/



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why are both locals() and globals() set?

[Steve D'Aprano]

On Sat, 23 Dec 2017 03:01 pm, Peng Yu wrote:

> Hi, The following example shows that both locals() and globals() are
> updated when x and f are defined. Shouldn't they be considered and
> global variable and functions only? Why does it make sense to set
> locals() as well? Thanks.

There are three places you can call locals() and globals():

(1) Inside a function:

def function():
x = 1
L = locals()
G = globals()
print "id(L)=%d  id(G)=%d" % (id(L), id(G))


(2) Inside a class definition:

class Class(object):
x = 1
L = locals()
G = globals()
print "id(L)=%d  id(G)=%d" % (id(L), id(G))

(3) At the top level of a module (outside of any function or class):

# module
x = 1
L = locals()
G = globals()
print "id(L)=%d  id(G)=%d" % (id(L), id(G))



If you try this, you will find that in #3, the dicts L and G have the same ID
number. This means that at the module level, locals() and globals() are the
same dict. This is (barely!) documented here:

https://docs.python.org/2/reference/executionmodel.html#binding-of-names

where it tells us that "The variables of the module code block are local and
global" but not documented here:

https://docs.python.org/2/library/functions.html#locals

I've raised an issue for that too: https://bugs.python.org/issue32413


Also note that for technical reasons, assigning to locals() inside a function
is not guaranteed to always work. The locals() dict returned is *sometimes* a
copy of the actual local variables, and so assigning to it does not assign to
the actual local variable. This is a tricky area of Python, where different
versions and different implementations (such as CPython, IronPython, Jython,
PyPy) may behave differently.


By the way, using globals() and locals() is considered an "advanced" feature.
If you want to set a variable, use ordinary assignment:

# set a local
x = 1

# to set a global from inside a function or class, declare it global
global x
x = 1

rather than trying to play around with globals() and locals(). 



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Where is ^ (symmetric_difference) documented in help()?

[Steve D'Aprano]

On Sat, 23 Dec 2017 03:50 pm, Peng Yu wrote:

> Where is it documented that __xor__ and ^ is the same as
> symmetric_difference? Thanks.

You read 

https://docs.python.org/2/library/stdtypes.html#set

to learn that ^ is the same as symmetric difference, and then read:

https://docs.python.org/2/reference/datamodel.html#emulating-numeric-types

to learn that you must override __xor__ to override the ^ operator.

It isn't really clear from the documentation that the set operator ^ is
implemented by __xor__ (and also __rxor__). Perhaps you can suggest a
documentation patch?



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Where is ^ (symmetric_difference) documented in help()?

[Steve D'Aprano]

On Sat, 23 Dec 2017 02:35 pm, Peng Yu wrote:

> Hi, I see the following two lines are the same. But I'd like to find
> where ^ is documented via the help() function (I am not looking for
> the document in html)? Does anybody know? Thanks.
> 
> s.symmetric_difference(t)
> s ^ t


You can call:

help("^")

(notice you need to quote the operator) but it only talks about the bitwise
operators.


I've just raised bug report for this: 

https://bugs.python.org/issue32412




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why "flat is better than nested"?

[Steve D'Aprano]

On Sat, 23 Dec 2017 01:48 am, andrewpat...@gmail.com wrote:

> On Monday, October 25, 2010 at 11:07:42 AM UTC+1, kj wrote:
>> In "The Zen of Python", one of the "maxims" is "flat is better than
>> nested"?  Why?  Can anyone give me a concrete example that illustrates
>> this point?
>> 
>> TIA!
>> 
>> ~kj
>> 
>> PS: My question should not be construed as a defense for "nested".
>> I have no particular preference for either flat or nested; it all
>> depends on the situation; I would have asked the same question if
>> the maxim had been "nested is better than flat".
> 
> I think there is a point where flat stops working. One of the products I
> work on has a 40-50 field datastructure - it is hard to work with and find
> the appropriate fields in. So I would say structured is better than flat but
> simple is better than to structured.


Do you realise you are responding to a message more than seven years old?

In any case, structured is not the opposite of flat. Nested is the opposite to
flat, and unstructured is the opposite of structured. The two concepts are
independent of each other: any data structure can be:

- flat and structured;
- flat and unstructured;
- nested and structured;
- nested and unstructured.

Data can even be semi-structured; for example, mp3 files have some structured
metadata (title, artist, track number, comment, etc), while data in the
comment field itself is unstructured (free-form) text.


Here is an example of iterating over a flat collection of numbers:

values = [1, 2, 3, 4, 5, 6, 7, 8]
for num in values:
print(num)


Here is the same, as a nested collection of numbers:

values = [1, [2, [3, [4, [5, [6, [7, [8, []
while values:
num, values = values
print(num)


In Python, the first is much more efficient than the second. It is also easier
to write, easier to read, and less likely for the programmer to mess up.

A list or a dict is flat; a binary tree is nested.

Structured and unstructured data can have two related but separate meanings.
One is to distinguish between data with or without a pre-defined
organization:

- structured data: XML, JSON, YAML, databases, etc;

- unstructured data: books, the body of emails, arbitrary web pages, etc.

Dealing with unstructured data in this sense often means coming up with some
sort of heuristic or "best guess" for picking out the useful information
(say, based on regexes) then writing a scraper to pull part the data source
looking for what you want.


The meaning you seem to be using seems to be:

- structured data has named fields (e.g. C struct, Pascal record, 
  object with named attributes, Python named tuple, CSV file with 
  descriptive column headers);

- unstructured data does not name the fields (e.g. a plain tuple,
  CSV file without column headers) and you have to infer the 
  meaning of each field from out-of-band knowledge (say, you read
  the documentation to find out that "column 7" is the score).


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to edit a function in an interactive python session?

[Steve D'Aprano]

On Thu, 21 Dec 2017 12:42 pm, Peng Yu wrote:

> Hi,
> 
> R has the function edit() which allows the editing of the definition
> of a function. Does python have something similar so that users can
> edit python functions on the fly? Thanks.
> 
> https://www.rdocumentation.org/packages/utils/versions/3.4.3/topics/edit

No.

In the standard interactive interpreter, you can only re-enter the function,
either re-typing it, or using the readline "history" to call up each line
again. (This may not work under Windows.)

iPython or Jupyter may offer extra editing functionality.

Or you could try creating your own. This might be a good place to start:

https://mail.python.org/pipermail/python-list/2014-September/677841.html

http://code.activestate.com/recipes/578926-call-out-to-an-external-editor/

 
Read the full thread -- it has lots of useful information.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: correctness proof for alpha-beta algorithm

[Steve D'Aprano]

On Thu, 21 Dec 2017 08:37 am, Bill wrote:

> namenobodywa...@gmail.com wrote:
>> On Tuesday, December 19, 2017 at 3:28:39 PM UTC-8, Steve D'Aprano wrote:
>>
>>> Does this have anything specifically to do with Python programming?
>> i'm working on a game-playing script (ie: in python), i want to incorporate
>> pruning into my search algorithm, and i'd like to understand why it works;
>> i'm not sure if that counts
> 
> Based upon your posts, I think you should just write your own. Then you
> can be sure that it will work. 

How long did you say you've been programming? Perhaps you've heard of
something we in the programming world call "bugs".




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

[META] Are the list admins honouring Posting Prohibited demands?

[Steve D'Aprano]

This is possibly a question for the list admins...

I notice that Lawrence D’Oliveiro has taken up labelling his posts with a
demand that his posts are not to be posted to the Python-List mailing list.

I also see that his posts are not showing up on the mailing list archive. Is
this a coincidence or a direct consequence of his demand?


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: correctness proof for alpha-beta algorithm

[Steve D'Aprano]

On Wed, 20 Dec 2017 07:23 am, namenobodywa...@gmail.com wrote:

> On Monday, December 18, 2017 at 10:16:07 PM UTC-8, Terry Reedy wrote:
> 
>> Where or how have you looked so far?  How formal do you want?
> 
> i want full-on formal with lots of rigor and every possible detail spelled
> out; i've looked in a couple of books but my best lead so far is a paper by
> knuth called "an analysis of alpha-beta pruning" - what i need is along
> those lines but with just a few more of the details spelled out

Does this have anything specifically to do with Python programming?

If not, why are you asking here? Do you think that Python programmers are
especially well-known for their hard-core formal academic methodology?

Have you tried searching on Google Scholar?

We're pretty accepting of off-topic posts here, especially when they evolve
naturally from an on-topic post. But in future, if you're going to *start* an
off-topic thread from the first post, it would be polite to label it such
with an "[OT]" or "Off-topic" prefix to the subject line.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with timeit

[Steve D'Aprano]

On Tue, 19 Dec 2017 02:27 am, ast wrote:

> I discovered that log functions from math module
> works with integers, whatever their size, there is
> no conversion to float.
> 
>> import math
>> x = 123456**123456
>> math.log10(x)
> 628577.7303641582   (instantaneous)
> 
> so 628578 digits


Nice!

It also works with other bases:

py> x = 10**100
py> math.log(x, 2)
3321928.0948873623
py> x.bit_length()
3321929


Today I learned. Thank you.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with timeit

[Steve D'Aprano]

On Sat, 16 Dec 2017 12:25 am, ast wrote:

> 
> "Thomas Jollans"  a écrit dans le message de
> news:mailman.74.1513341235.14074.python-l...@python.org...
>> On 2017-12-15 11:36, ast wrote:
> 
> 
>> No, this is right. The calculation takes practically no time; on my
>> system, it takes some 10 ns. The uncertainty of the timeit result is at
>> least a few hundred nanoseconds.
> 
> There are more than 10 multiplications to perform on a
> soaring size integer. I have some doubts "x=123456**123456 "
> only takes 10 ns on your system.

You would be right.

My computer appears to be about a thousand times slower than Thomas' computer.
On my computer, it takes about a second to calculate 123456**123456, so I
predict his will take about a millisecond.


> On my computer it takes roughtly 4 s, mesured with a watch.

That's not a very accurate way to measure it. You are measuring your own
reaction time, the time it takes the interactive interpreter to parse the
text of the code, compile it, execute the code, then print a new prompt, and
then your reaction time again. The actual execution time is drowned in the
rest of the noise.

> I can't do "len(str(x))" to know the size, I have to kill the process

On my computer, that took about five minutes or so. I wasn't really paying
attention on precisely how long it took, but it was around that.


> But x.bit_length() answers 2088091, so x should have about
> 60 digits

py> x = 123456**123456
py> s = str(x)
py> len(s)
628578



> If I measure execution time:
> 
 t=time(); x=123456**123456; print(time()-t)
> 0.0
> 
> There is still something wrong

The interpreter first computes the big int 123456**123456 while it is
compiling the entire command line, so that happens *before* the time starts.
The assignment is so fast that the three statements:

t = time()
x = 5...6  # huge number, pre-calculated
print(time() - t)

essentially occurs faster than the resolution of time.time() on your machine.

Try using time.perf_counter instead.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with timeit

[Steve D'Aprano]

On Fri, 15 Dec 2017 10:47 pm, Thomas Jollans wrote:

> On 2017-12-15 11:36, ast wrote:
>> Hi
>> 
>> Time measurment with module timeit seems to work with some statements
>> but not with some other statements on my computer.
>> 
>> Python version 3.6.3
>> 
>> from timeit import Timer
>> 
> Timer("'-'.join([str(i) for i in range(10)])").timeit(1)
>> 0.179271876732912
> Timer("'-'.join([str(i) for i in range(10)])").timeit(10)
>> 1.7445643231192776
>> 
>> It's OK, with 10 more loops I get 10 more execution time.
>> 
>> But with exponentiation, it's a mess
>> 
> Timer("x=123456**123456").timeit(1)
>> 6.076191311876755e-06
> Timer("x=123456**123456").timeit(10)
>> 3.841270313387213e-06
>> 
>> All wrong, the calculation of 123456**123456 is much longer
>> than 6 microseconds, it takes few seconds, and with 10 loops timeit
>> provided a shorter time ...
> 
> No, this is right. The calculation takes practically no time; on my
> system, it takes some 10 ns. The uncertainty of the timeit result is at
> least a few hundred nanoseconds.

You've misdiagnosed Ast's problem, and fallen for the same Gotcha he has.
You're not measuring what you think you measured:

 Timer("x=123456**123456").timeit(10**6)
> 0.00969144597183913

Calculating BigInt exponentiation is fast, but its not that fast once you get
to hundreds of thousands of digits.

That calculates the 600,000+ digit number 123456**123456 once, at compile
time, then simply assigns that huge number to x a million times. So you would
likely get almost the same result by running:

Timer("x=17").timeit(10**6)




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with timeit

[Steve D'Aprano]

On Fri, 15 Dec 2017 09:36 pm, ast wrote:

[...]
> It's OK, with 10 more loops I get 10 more execution time.
> 
> But with exponentiation, it's a mess
> 
 Timer("x=123456**123456").timeit(1)
> 6.076191311876755e-06
 Timer("x=123456**123456").timeit(10)
> 3.841270313387213e-06
> 
> All wrong, the calculation of 123456**123456 is much longer
> than 6 microseconds, it takes few seconds, and with 10 loops
> timeit provided a shorter time ...
> 
> What happens plz ?


You are not measuring what you think you are measuring. Import the dis command
to disassemble your source code and see what you are measuring. Here is an
example:

py> from dis import dis
py> dis("x=123**123")
  1   0 LOAD_CONST   2
(1143743679346171900998802952280662767462180784518502297758879750523695
04785666896446606568365201542169649974727730628842345343196581134895919
94282087444983721209947664895835902379607854904194900780722062535652692
6729664064846685758382803707100766740220839267)
  3 STORE_NAME   0 (x)
  6 LOAD_CONST   1 (None)
  9 RETURN_VALUE

I simplified the expression to 123**123 instead of 123456**123456, in order to
make the disassembly small enough to read. Otherwise there would be thousands
of digits.

The important thing is that Python's keyhole optimizer is optimizing the
constant expression 123456**123456, calculating it at compile time, so you
are timing the equivalent of:

x = 5374822891...2343545856 # a 628578 digit number


All the expensive work is done once, at compile time (and even that only takes
a second or so). What timeit measures is just repeatedly assigning x to the
pre-defined constant, over and over again.

So in theory, the speed should be **really really fast**. In practice, you are
measuring more-or-less just the overhead of assignment, and the overhead of
timeit itself. That overhead is so small that it is dominated by the random
noise of your computer, which is doing many other things at the same time,
and it is just a fluke that sometimes calling timeit(10) will be shorter than
calling timeit(1).

On my computer, I get results like this:


py> Timer("x=123456**123456").timeit(1)
7.5660645961761475e-06
py> Timer("x=123456**123456").timeit(1)
6.902962923049927e-06
py> Timer("x=123456**123456").timeit(1)
5.256384611129761e-06

The fact that the time taken keeps getting smaller might be a fluke, or it
might have something to do with CPU prediction and caching results. Now look
what happens when I increase the number of iterations:


py> Timer("x=123456**123456").timeit(10)
7.111579179763794e-06
py> Timer("x=123456**123456").timeit(10)
5.822628736495972e-06
py> Timer("x=123456**123456").timeit(10)
5.457550287246704e-06

The time barely changes. My prediction is that this is because the time of the
actual assignment is so small, the measured time is dominated by timeit
itself. I can test this:

py> Timer("pass").timeit(1)
4.108995199203491e-06
py> Timer("pass").timeit(10)
4.8354268074035645e-06


So that is approximately the fastest thing timeit can measure on my computer,
and there is no significant difference between doing it once, and doing it
ten times. The difference is lost in the noise.


The right way to measure this is like this:


py> Timer("x = n**n", "n = 123456").timeit(1)
0.9905387535691261
py> Timer("x = n**n", "n = 123456").timeit(10)
18.006236914545298


As you can see, repeating the calculation 10 times doesn't take exactly ten
times as long as doing the calculation once. That could be because the
garbage collector runs. The more expensive the calculation, the more overhead
there is. Running Python on a modern machine with CPU branch prediction and
caching isn't like running pure assembly on an old, single-process machine
where every run is 100% deterministic.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Please tell me how to execute python file in Ubuntu by double

[Steve D'Aprano]

On Fri, 15 Dec 2017 09:48 am, Gregory Ewing wrote:

> Rhodri James wrote:
>> Even then there was RiscOS, which divorced file names from file types
>> entirely.
> 
> As did classic MacOS.

Classic MacOS associated two such pieces of metadata with each file: the
creator and type. Regardless of the optional file type extension, you could
associate different creators with different files of the same type:

https://en.wikipedia.org/wiki/Creator_code


> I believe MacOSX also has the ability to store a file type
> as metadata, but it doesn't seem to be used much.

They are still supported (-ish) by OS X, but have been superseded by Uniform
Type Identifiers.


https://en.wikipedia.org/wiki/Uniform_Type_Identifier



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: unabe to import /pyd file.

[Steve D'Aprano]

On Fri, 15 Dec 2017 09:09 pm, Tim Golden wrote:

> Apart from anything else these need to be raw strings:
> 
> sys.path.append(r'C:\Python27\Lib\lib-tk')

Don't use raw strings for paths. It's a trap:


r'C:\Python27'  # okay

r'C:\Python27\'  # fails


Windows supports / as directory separator. You should use / as the directory
separator in your source code, and let the OS convert them to backslashes.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: write your replace function in python

[Steve D'Aprano]

On Thu, 14 Dec 2017 09:08 pm, ayaskant.mantu...@gmail.com wrote:

> Hi,
> 
> I want to replace the spaces in a sting with hyphen with my own replace
> function or with using the pre-defined replace function. Can anybody help me
> with this issue???

new_string = "string with spaces".replace(" ", "hyphen")


When in doubt, open the Python interpreter to a command prompt ">>> " and
enter:

help("")

to see the string methods. Other commands work too:

help(str.replace)




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Please tell me how to execute python file in Ubuntu by double clicking on file. (Posting On Python-List Prohibited)

[Steve D'Aprano]

On Mon, 11 Dec 2017 11:29 am, Chris Angelico wrote:

> On Mon, Dec 11, 2017 at 10:10 AM, Rick Johnson
>  wrote:

>> And it's not like we can just pick file up and shake
>> it, in a crude attempt to intuit the contents.
> 
> No, but fortunately we have magic. And magic can tell us a lot about
> what a file contains.
> 
> man 5 magic

https://pypi.python.org/pypi/python-magic



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Save and load initialized class

[Steve D'Aprano]

On Sun, 10 Dec 2017 04:52 am, MRAB wrote:

> Try updating __dict__:
> 
>  Opts.__dict__.update(json.load(open("mybuffer")))

__dict__ is implementation, vars() is the public interface:

vars(Opts).update(json.load(open("mybuffer")))


Looks nicer too :-)




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Please tell me how to execute python file in Ubuntu by double clicking on file. (Posting On Python-List Prohibited)

[Steve D'Aprano]

On Sun, 10 Dec 2017 02:01 pm, Chris Angelico wrote:

> On Sun, Dec 10, 2017 at 12:56 PM, Steve D'Aprano
> <steve+pyt...@pearwood.info> wrote:
>> Remember the context here: we're replying to a thread discussing somebody
>> who is running Ubuntu with a GUI desktop environment. Of course there are
>> *some* Linux systems which don't run a GUI at all, but you can't
>> double-click on files on such systems, and they aren't Ubuntu, so they
>> aren't relevant.
> 
> Been a long time since I had an Ubuntu, but is it really the case that
> you can't install Ubuntu without a GUI?

I mispoke -- I meant something like "they aren't whatever Ububtu GUI the OP
has installed".

There is, apparently, headless Ubuntu, but it isn't clear to me whether they
install X (or Mir, Wayland or some other alternative) or not. E.g. this:

https://www.howtoforge.com/tutorial/ubuntu-gnome-vnc-headless-server/

seems to suggest to me that X.org is already installed, I can see X utils
being installed as a recommended package but I couldn't spot X being added as
a dependency. So my *guess* is that even headless Ubuntu server includes an X
server. (Or possibly Mir.)



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Please tell me how to execute python file in Ubuntu by double clicking on file. (Posting On Python-List Prohibited)

[Steve D'Aprano]

On Fri, 8 Dec 2017 12:08 pm, Python wrote:

> But more importantly, practically speaking, it still doesn't really
> provide much more help to the OP than Lawrence's answer.

I wasn't responding to the OP, I was responding to Lawrence. If I had a
solution for the OP beyond what others have already said (especially Thomas
Jollans' link to an AskUbuntu post), I would have given it.

Practically speaking, your responses to me don't help the OP either, do they?
So why single me out for criticism for *exactly* the same thing that you
yourself are doing?


> He may well 
> know already that the desktop environment is what does the job (and
> probably does even, in broad terms, if he's familiar with computers in
> general), but have no idea how to configure it.

Anything is possible, but if he were that clueful, he would know that this has
nothing to do with Python and he should be asking in a forum dedicated to his
preferred desktop environment. The fact that this is a Python script is
irrelevant.


> A reasonably helpful 
> answer would be one that mentioned a few of the likely possibilities
> (Gnome, KDE, Unity, /etc/mime.types, "other"), and gave hints for how
> to find out the answer for each.  A thoroughly helpful answer would
> be, well, outside the scope of this list/group.

Arguably -- and I'm not sure that I personally would take this position -- it
might be said that somebody clueless enough to ask a desktop-specific
question without specifying which desktop he is running, would only be
confused rather than enlightened by an answer which lists three or more
desktops.

If we take a user-centric position, there's a lot to be said for the
attitude "Don't bother me with technical details, I just want to make the
file executable when I double-click on it".

(I presume that there's a single, trivial, right answer to this question on
Mac OS and Windows.)


> Pedantry has its place, FWIW. In the computer field, as with other
> science and engineering disciplines, often precision is much more
> essential than in other fields.  I personally find such precision is
> especially warranted if you take it upon yourself to criticize what
> someone else has said.

There's a famous story where some English Lit student took it upon themselves
to criticise Isaac Asimov for his claim to be living in a century where we
finally knew the basics of how the universe worked. Asimov's response was to
quote something he had once told his editor, John Campbell:

[W]hen people thought the earth was flat, they were wrong. When people
thought the earth was spherical, they were wrong. But if you think that
thinking the earth is spherical is just as wrong as thinking the earth
is flat, then your view is wronger than both of them put together.

http://hermiene.net/essays-trans/relativity_of_wrong.html

https://en.wikipedia.org/wiki/Wronger_than_wrong

(This is becoming ever more relevant again, as for some reason -- possibly
Poe's Law -- the number of people on the Internet claiming to believe the
world is flat is on the rise.)

If you think that my statement is just as wrong as Lawrence's statement,
you're wronger than both of us together :-)


Remember the context here: we're replying to a thread discussing somebody who
is running Ubuntu with a GUI desktop environment. Of course there are *some*
Linux systems which don't run a GUI at all, but you can't double-click on
files on such systems, and they aren't Ubuntu, so they aren't relevant.


> Though, providing such precision via natural 
> language often turns out to be more challenging than one would hope...

Indeed.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Benefits of unicode identifiers (was: Allow additional

[Steve D'Aprano]

On Sat, 9 Dec 2017 09:57 am, Gilmeh Serda wrote:

> And next demands to allow Unicode as keywords in a translated version of
> Python
>  will make open source go away. For good.


Do you seriously think that because *one* project forks their code base and
introduces non-English keywords, the tens of hundreds of thousands of other
projects (including *enormous* projects like the Linux kernel, LibreOffice,
Apache, Mozilla, etc) will all say "well, that's it, we're shutting down"?



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Benefits of unicode identifiers (was: Allow additional separator in identifiers)

[Steve D'Aprano]

On Sun, 10 Dec 2017 09:20 am, Terry Reedy wrote:

> On 12/9/2017 5:57 AM, Gilmeh Serda wrote:
> 
>> And next demands to allow Unicode as keywords in a translated version of
>> Python
> 
> Python's liberal open source license allows people to revise and
> distribute their own python or python-like interpreters.  I believe
> there are already a couple of non-english versions aimed at schoolkids.
> The cpython core developer group has nothing to do with such.

I don't know who their target audiences are, but there's a German and Chinese
version of Python.

http://www.fiber-space.de/EasyExtend/doc/teuton/teuton.htm

http://www.chinesepython.org/english/english.html

My *guess* is that Teuton is intended as a proof-of-concept just to show it
can be done, and ChinesePython is intended for students.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: f-string

[Steve D'Aprano]

On Wed, 6 Dec 2017 11:54 am, John Pote wrote:

[...]
> Ran above test file and got,
>  >>python36 compiletest.py
>  at 0x02120E40, file "", line 1>
>
>
> SPAM scrambled

Thanks everyone, that's what I wanted to see.



--
Steve
â £Cheer up,â Ø they said, â £things could be worse.â Ø So I cheered up, and
sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: f-string

[Steve D'Aprano]

On Wed, 6 Dec 2017 12:21 pm, Chris Angelico wrote:

> On Wed, Dec 6, 2017 at 11:54 AM, John Pote <johnp...@jptechnical.co.uk>
> wrote:
>>
>> On 06/12/2017 00:16, Steve D'Aprano wrote:
>>>
>>> Anyone got a handy copy of Python 3.6 available to test something for me?
>>>
>>> What does compile('f"{spam} {eggs}"', '', 'single') return?

[...]

> I think Steve just wanted to see what we'd all define spam and eggs as.
>
> ChrisA
> *ducking for cover*

Possibly duck eggs :-)


I was thinking of responding to your comment on Python-Ideas that said there is
 no way to represent an unevaluated f-string in Python, but then I decided that
 even for me that was too pedantic.




--
Steve
â £Cheer up,â Ø they said, â £things could be worse.â Ø So I cheered up, and
sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: we want python software

[Steve D'Aprano]

On Wed, 6 Dec 2017 03:45 pm, Abhiram R wrote:

> On Wed, Dec 6, 2017 at 10:08 AM, km  wrote:
>
>> I dont know how these students are selected into b tech stream in India.
>> they are so dumb. All they know is a to open a program we need to double
>> click it and it runs.
>>
>> We were all once "dumb". We learnt it because someone Taught us.


Since you're so open to being corrected, let me remind you to read your posts
before sending. Somehow you managed to introduce an extra > quote marker at the
 beginning of your own comment, (the line starting "We were all once...").
Check it out here:

https://mail.python.org/pipermail/python-list/2017-December/729112.html

and you will see that your comment is quoted, as if it were written by the
person you are replying to.

(Normally I wouldn't bother mentioning something so trivial, but since you make
 the excellent point that we don't learn to correct mistakes unless we have
them pointed out, I thought I'd do so.)


> I'd rather not entertain such or refrain from condescending replies

I think you have misunderstood the word "condescending".

It is condescending to assume that the OP, Jyothiswaroop Reddy, is a fragile
and delicate little hothouse flower that needs protecting from reality where
people will tell you "Don't be so lazy and don't waste our time".

I prefer to expect more of people and let them meet my expectations, than to
downgrade my expectations and coddle them into a spiral of lower and lower
competence and ability.

(By the way Rustom, if you're reading, thank you for that link to the video a
few weeks ago about teaching 2 + 2 = 22. My blood pressure just about doubled
watching it.)




--
Steve
â £Cheer up,â Ø they said, â £things could be worse.â Ø So I cheered up, and
sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: csv.DictReader line skipping should be considered a bug?

[Steve D'Aprano]

On Wed, 6 Dec 2017 04:20 am, Jason wrote:

> I ran into this:
>
https://stackoverflow.com/questions/27707581/why-does-csv-dictreader-skip-empty
-lines
>
> # unlike the basic reader, we prefer not to return blanks,
> # because we will typically wind up with a dict full of None
> # values
>
> while iterating over two files, which are line-by-line corresponding. The
> DictReader skipped ahead many lines breaking the line-by-line
> correspondence.

Um... this doesn't follow. If they are line-by-line corresponding, then they
should skip the same number of blank lines and read the same number of
non-blank lines.

Even if one file has blanks and the other does not, if you iterate the over the
 records themselves, they should keep their correspondence.

I'm afraid that if you want to convince me this is a buggy design, you need to
demonstrate a simple pair of CSV files where the non-blank lines are
corresponding (possibly with differing numbers of blanks in between) but the
CSV readers get out of alignment somehow.


> And I want to argue that the difference of behavior should be considered a
> bug. It should be considered as such because: 1. I need to know what's in
> the file to know what class to use.

Sure. But blank lines don't tell you what class to use.

> The file content should not break at-least-1-record-per-line.

Blank lines DO break that requirement. A blank line is not a record.


> There may me multiple lines per record in the
> case of embedded new lines, but it should never no record per line.

I disagree. A blank line is not a record. If I have (say) five fields, then:

\n

is a blank record with five empty fields. \n alone is just a blank. The
DictReader correctly returns records with blank fields.


> 2.  It's a premature optimization. If skipping blank lines is desirable,
> then have another class on top of DictReader, maybe call it
> EmptyLineSkippingDictReader.

No, that's needless ravioli code. The csv module already defines a basic reader
 that doesn't skip blank lines. Having two different DictReaders, one which
doesn't work correctly because it wrongly expands blank lines to collections of
 blank fields, is not helpful.

Perhaps if they were called BrokenDictReader for the one which expands blank
lines to empty records, and DictReader for the one which correctly skips blank
lines.


> 3. The intent of DictReader is to return a
> dict, nothing more, therefore the change of behavior isn inappropriate.

No, if all you want is a dict, call dict() or use the dict display {}. The
intent of DictReader is to *read a CSV file and extract the records* as a dict.
 Since blank lines aren't records, they should be skipped.


> Does anyone agree, or am I crazy?

I wouldn't want to guess your mental health based just on this isolated
incident, but if I had to make a diagnosis, I'd say, yes, crazy as a loon.

*wink*




--
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Please tell me how to execute python file in Ubuntu by double click

[Steve D'Aprano]

On Tue, 5 Dec 2017 07:58 pm, Lawrence Dâ ÖOliveiro wrote:

> On Tuesday, December 5, 2017 at 3:39:26 AM UTC+13, Rick Johnson wrote:
>>
>> Sounds like your OS file associations are all botched-up ...
>
> Linux doesnâ Öt do â £OS file associationsâ Ø.


Then how does my Linux box know that when I double-click on a text file, it
launches kwrite rather than (say) the Gimp or LibreOffice?

When I right-click on a mp4 video, I get a menu that includes a Open With
command that shows (amount others) Kaffeine, mplayer and VLC.

If you mean the Linux *kernel* doesn't do file associations, then you should
have said so.

But why do you care about the kernel? Would you think it even the *tiniest*
useful to claim that "Linux doesn't do email" because it is sendmail or postfix
 (or similar) that sends email rather than the Linux kernel itself?



--
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

f-string

[Steve D'Aprano]

Anyone got a handy copy of Python 3.6 available to test something for me?

What does compile('f"{spam} {eggs}"', '', 'single') return?

What does eval()'ing the above compiled object do? If necessary, you may have
to define spam and eggs first.


Thanks in advance.


--
Steve
â £Cheer up,â Ø they said, â £things could be worse.â Ø So I cheered up, and
sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Round to 2 decimal places

[Steve D'Aprano]

On Thu, 7 Dec 2017 01:31 pm, nick martinez wrote:

> interesting, what version of python are you using? Tried it multiple times
> and it still isn't working.


Please launch a terminal window, copy this command exactly into the terminal
and hit ENTER. You should have a $ or maybe % prompt for this to work.

python3.5 -c "volume = 257.148345678; print(round(volume, 2))"

(Do you need help opening a terminal window? If so, ask. We'll need to know
your operating system, and if you are using Linux, which desktop environment
you are using.)

If your prompt is ">>>" (without the quotes) or something similar to "In [1]:"
then copy these two lines to the prompt and hit ENTER:

volume = 257.148345678
print(round(volume, 2))

Either way, reply here with the output. Make sure you copy it.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Round to 2 decimal places

[Steve D'Aprano]

On Thu, 7 Dec 2017 11:58 am, nick martinez wrote:

> I'm stuck. I need my program to round the end solution to 2 decimal places
> but cant figure it out. Can someone help? I've been trying between printf
> and round() but cant seem to get either to work.

It might help if you show exactly what values you have tried. This works for
me:

py> volume = 1.23456789
py> print(volume )
1.23456789
py> print(round(volume , 2))
1.23

The rest of your code is interesting but irrelevant to your question. If you
want to learn how to round to 2 decimal places, the actual calculation of
surface area and volume aren't necessary. The only really important part is
this line:

>print("A cone with radius", r, "\nand height of", h, "\nhas a volume of :
>", volume, "\nand surface area of : ", surfacearea,)


and even that can be cut down to just:

volume = 2345.987654321  # for example
print(volume)


We can round the result first, then print it:


print(round(volume, 2))


or we can change the display by using string formatting:


print("%.2f" % volume)

print("{:.2f}".format(volume))



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Please tell me how to execute python file in Ubuntu by double clicking on file. (Posting On Python-List Prohibited)

[Steve D'Aprano]

On Thu, 7 Dec 2017 08:22 am, Python wrote:

> On Wed, Dec 06, 2017 at 10:35:58AM +1100, Steve D'Aprano wrote:
>> On Tue, 5 Dec 2017 07:58 pm, Lawrence D’Oliveiro wrote:
>> 
>> > On Tuesday, December 5, 2017 at 3:39:26 AM UTC+13, Rick Johnson wrote:
>> >> 
>> >> Sounds like your OS file associations are all botched-up ...
>> > 
>> > Linux doesn’t do “OS file associations”.
>> 
>> 
>> Then how does my Linux box know that when I double-click on a text file, it
>> launches kwrite rather than (say) the Gimp or LibreOffice?
> 
> The answer to that is (sadly) complicated.

Actually, no, the answer to my question is very simple: Lawrence is mistaken
about Linux not doing file associations. It does -- it is merely handled by
the desktop environment (if there is one).

Lawrence is showing off his geek cred with an answer that is pedantically
correct but in practice not just useless but actually false. Of course Linux
computers can associate certain files with certain applications.

If the term "OS file associations" is ever so slightly inaccurate (it's not
the actual OS kernel that does the associating, but the desktop environment),
well, we can probably say the same thing about Mac OS X and maybe even
Windows itself.


> How it knows indeed 
> depends on the context of the click, what desktop environment you're
> using, what application(s) you're using, and what configurations
> you've made to those things to address any file associations.

Indeed you are correct, which is why it may not be a trivial matter to answer
the OP's question about how to get the behaviour he wants. But that isn't
what my comment is about. I'm simply replying to Lawrence's insinuation that
Linux computers don't associate file types to applications.

I'm sure that Lawrence worded his statement *ever so carefully* to ensure that
it was pedantically correct if read to the letter, while still insinuating
something completely wrong. It takes a lot of care to provide information
which is both true and utterly misleading at the same time, but Lawrence does
it very, very well indeed.

If only he would use his considerable intellect to provide *useful* answers
instead of *accurate but misleading* answers.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Module _socket not found in python3.6 "No module named _socket"

[Steve D'Aprano]

On Thu, 7 Dec 2017 07:59 am, Bryan Zimmer wrote:

> I have been getting this message, "No module named '_socket'", since I
> installed python 3.6, about two months ago.
> 
> My platform is Slackware Linux (14.2). I compiled python3.6 from source,
> because binary python packages aren't distributed by python.org for Linux.
> I have the same experience on multiple Slackware computers, not just one.
> Actually, this problem hits the 64-bit distribution. I have a 32-bit
> Slackware box that has no trouble with _socket.

You need to look at the output of compilation. That might mean doing it
again :-(

Unfortunately the default is to output everything including the kitchen sink,
which makes it hard to notice the errors.

Take note of which files failed to compile, and if need be, post the list
here. Chances are you are missing some development library. You may have the
32-bit dev library installed, but not the 64-bit version.


> I can not 'import socket' into a program, nor can I use 'pip' to install
> modules. These require the _socket module, which doesn't exist. What does
> exist is a shared library object called '_
> socket.cpython-36m-i386-linux-gnu.so'. This looks tantalizingly like the
> missing module, but it is for a 32-bit architecture.

Where is that file? On your 64-bit box?

Do you have the 32-bit Python 3.6 already installed on that machine? If so,
make sure you install the 64-bit version in a different location.

If this isn't enough to diagnose the problems, best to go back to basics and
post *each* step you take and any error output. (The config and make steps
tend to send a metric ton of uninformative output to stdout, probably best to
just capture stderr.) Trivial steps that you are sure are correct can be
summarised (e.g. "Downloaded Python3.6.tar.gz, unpacked it, and cd'ed into
the resulting directory") but if you have any doubt that you are doing it
correctly, better to post too much information than not enough.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: we want python software

[Steve D'Aprano]

On Wed, 6 Dec 2017 04:54 pm, Chris Angelico wrote:

> On Wed, Dec 6, 2017 at 4:27 PM, km  wrote:
>> Remember that you are wasting time of lakhs of  python subscribers by
>> asking such dumb questions being tech students.  You people can Google and
>> watch movies / songs online  and you can't find how to download and install
>> python ? That's ridiculous!
>>
> 
> This attack is unwarranted. Please don't set fire to people simply
> because they asked a question like this.

I'm going to defend KM (srikrishnamohan) -- his comments were not "an attack",
they are a well-deserved criticism of a *tech student* who apparently made
zero effort to find out how to download Python before asking others to do it
for him.

I'm sorry for the length of this post. It is trivially easy to sink the boots
in and tell KM off for his blunt criticism of the OP's request. But in the
face of this hostile environment (out of the nine people who responded to
this thread, no fewer than three have piled onto KM to tell him off), it
isn't so easy to get through the message of why we shouldn't always coddle
people asking questions like that asked by the OP, and why KM's response was
tough but fair.

It's not enough to merely shout back "No, you're wrong!", hence the length of
this reply.


Its been a while since I've seen anyone here link to "How to ask questions the
smart way":

http://catb.org/~esr/faqs/smart-questions.html


and quote this:


When you ask your question, display the fact that you have done
these things [try to find an answer] first; this will help establish
that you're not being a lazy sponge and wasting people's time.


(And more importantly: *you might learn something* by trying to solve your own
problem.)

And yes, I'm aware of the irony of me taking this position only a couple of
posts after I asked the group to run some code for me without explaining why
I couldn't run it myself.[1]

The difference is, I have many years of answering other people's questions,
I'm well known here, and I have a proven track record of not being a lazy
sponge. But if somebody wants to take me to task for not explicitly stating
why I wasn't running the code myself, I will preemptively take it in good
grace and accept the criticism. Mea culpa.

How likely is it that somebody who is tech-savvy enough to sign up and post to
the Python-List mailing list is not savvy enough to have heard of google or
to have thought of search terms "download Python"?

https://duckduckgo.com/html/?q=download%20python

We're not talking about a young child, or some other barely computer literate
person, but somebody studying for a Bachelor of Technology in India.

Offering criticism is not attacking somebody. Not even tough criticism.

And if you are thinking that it is, well, consider the beam in your own eye
before the mote in KM's. You have just "attacked" (criticised) KM quite
harshly, accusing him of making an "unwarranted" attack (I think it was very
warranted), and using a metaphor which has particular cultural and colonial
associations in India and neighbouring countries which we Westerners should
be wary of making without good cause.[2]

When somebody mildly breaches social norms, even the norms of a tech forum,
mild shaming is often an effective method of enforcement. I'm not saying that
the OP should be doxxed, his family and employer harassed, ripped apart on
social media, but KM telling him off for wasting people's time seems fair to
me. We can assume that the OP isn't a two year old. He should know better,
and we ought to expect more from him. Its not a crime if he doesn't, but we
don't have to molly-coddle him either.

Consider your bible: a soft answer turns away wrath. But the bible never says
that the wrath wasn't justified in the first place. KM is clearly angry at
the OP's behaviour, hence his strong words. We should balance our concern
about driving away newbies like the OP with some concern about the justified
anger at needy, entitled, demanding people who take, take, take and never
give back. "Smart Questions" (above) is not just good advice, it is also a
set of social norms, and the OP violated them.

And again, consider your own beam: what you are complaining about KM doing to
the OP, is exactly what you, Ethan and others are attempting to do to KM. You
consider KM's actions to have violated *your* social norm of "be nice" and
consequently you are trying to shame KM into changing his behaviour to meet
those norms, by chastising him and telling him off for supposedly attacking
the OP, for being condescending, for being hateful.

Ironically, it appears that both KM and the OP are newbies. Aren't we supposed
to be more welcoming to newbies?


> You can be far more courteous 
> than this, even if you refuse to help. Particularly, the "you people"
> sounds like a blanket statement, which is almost certainly not useful
> to the discussion.

I think that given both posters appear to be Indian, perhaps 

Re: Politeness (was: we want python software)

[Steve D'Aprano]

On Wed, 6 Dec 2017 11:25 pm, Rustom Mody wrote:

> On Wednesday, December 6, 2017 at 4:05:43 PM UTC+5:30, Steve D'Aprano wrote:
>> On Wed, 6 Dec 2017 02:49 pm, Rustom Mody wrote:
>> 
>> > You are assuming that the strangeness of the request is about 'tech'
>> > [engineering/tech existed centuries before computers]
>> > 
>> > Do remember one can be a tech-{student,professional} without
>> > - ever having encountered free-software
>> > - internet/USENET culture
>> > 
>> > … from which pov the request would not look so odd
>> 
>> So you're suggesting that rather than being unwilling to google for
>> "Download Python" because he doesn't understand free software culture, the
>> OP is unwilling to google for "Download Python" because he thinks it is
>> proprietary software and wants a bunch of strangers on the Internet to send
>> him a pirate copy?
>> 
>> I'm not entirely sure that's better...
> 
> Dunno anything about OP so no 'suggesting'…

Rustom, you LITERALLY suggested that the OP could be a tech student who isn't
familiar with free-software and that would explain the "strangeness" of the
request. What other interpretation of your words quoted above is reasonable?

You suggested that if he wasn't familiar with free software, his request that
people send him a copy of Python wouldn't look so odd. Okay, if Python
weren't free software, it would be non-free software, and you are saying that
it wouldn't look so odd for somebody to join a mailing list and request a
bunch of strangers to gift him a copy of non-free software.

That seems pretty odd to me. Laziness I get. Being so naive as to expect that
someone will purchase a legal copy and give it to a perfect stranger out of
the goodness of their heart, I don't think is so believable. But maybe that
happens all the time in India *shrug*

If the OP was as ignorant of software (whether free or not) and the Internet
as your defence of him makes out, I would expect his question would have
been "How do I get the Python? software" rather than "Send me the Python
software".

The OP clearly understands that Python is something that will run on his
computer, that it can be sent to him, that there are people willing and able
to do so, and he understands the internet well enough to expect those people
will be able to contact him without his needing to explicitly give them his
address.


> I did hint one point above which can be restated more explicitly.
> 
> An engineering degree (aka “B.Tech”) can be in any of
> - IT, CS, Computer Engineering etc
> - aeronautics, civil, electrical, mechanical… classical, non-computer
> related - bioinformatics, statistics, "scientific computing" etc ie
> heavy-duty *users*
>   of computers
> 
> For the latter two classes it would be normal/natural for the student to
> have little knowledge/interest in computer-related stuff except as a user.

None of this is the slightest bit relevant. The OP explicitly said that he
wants to learn Python. He's not some random civil engineering or biology
student, he is a tech student who wants to learn to program in Python.

And the idea that "users of computers" don't know how to use Google is
ludicrous. Google hasn't become one of the biggest tech companies in the
world because only a tiny number of Open Source programmers use it. It is
2017, not 1997, and Google is ubiquitous. Even my dad knows about Google and
he doesn't even have a computer, not even a smart phone.


[...]
> ie "unwilling to google" could well be "ignorant of google
> (usage/practices)" Do consider the possibility that a student could be a
> non-owner of a computer and/or studying in a college in a poor/non networked
> location.

Your hypothetical that the OP is a tech student who has barely used a computer
before and doesn't know about Google but nevertheless knows about Python
(which is about a million times less well-known) *and* is internet-savvy
enough to send an email to this mailing list is such a remote possibility
that I don't know why I'm even responding to it.


[...]
> IOW I would wish Ethan's "control yourself" to be imperated — preferably
> by oneself, if not then forcibly.
> And especially when the vitriol is flung at a first-time poster.

KM also appears to be a first-time poster. And after all the critical attacks
he's received, I doubt he'll hang around.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: csv.DictReader line skipping should be considered a bug?

[Steve D'Aprano]

On Thu, 7 Dec 2017 02:33 am, Dennis Lee Bieber wrote:

> On Wed, 06 Dec 2017 11:06:39 +1100, Steve D'Aprano
> <steve+pyt...@pearwood.info> declaimed the following:
> 
> 
>>I wouldn't want to guess your mental health based just on this isolated
>>incident, but if I had to make a diagnosis, I'd say, yes, crazy as a loon.
>>
> 
> Loons aren't crazy; they just sound that way. Kookaburras, OTOH...

Don't knock kookaburras. Anything that makes a living by eating Australian
snakes is worth respecting.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: we want python software

[Steve D'Aprano]

On Wed, 6 Dec 2017 02:49 pm, Rustom Mody wrote:

> You are assuming that the strangeness of the request is about 'tech'
> [engineering/tech existed centuries before computers]
> 
> Do remember one can be a tech-{student,professional} without
> - ever having encountered free-software
> - internet/USENET culture
> 
> … from which pov the request would not look so odd

So you're suggesting that rather than being unwilling to google for "Download
Python" because he doesn't understand free software culture, the OP is
unwilling to google for "Download Python" because he thinks it is proprietary
software and wants a bunch of strangers on the Internet to send him a pirate
copy?

I'm not entirely sure that's better...



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: we want python software

[Steve D'Aprano]

On Wed, 6 Dec 2017 03:45 pm, Abhiram R wrote:

> On Wed, Dec 6, 2017 at 10:08 AM, km  wrote:
> 
>> I dont know how these students are selected into b tech stream in India.
>> they are so dumb. All they know is a to open a program we need to double
>> click it and it runs.
>>
>> We were all once "dumb". We learnt it because someone Taught us.


Since you're so open to being corrected, let me remind you to read your posts
before sending. Somehow you managed to introduce an extra > quote marker at
the beginning of your own comment, (the line starting "We were all once...").
Check it out here:

https://mail.python.org/pipermail/python-list/2017-December/729112.html

and you will see that your comment is quoted, as if it were written by the
person you are replying to.

(Normally I wouldn't bother mentioning something so trivial, but since you
make the excellent point that we don't learn to correct mistakes unless we
have them pointed out, I thought I'd do so.)


> I'd rather not entertain such or refrain from condescending replies

I think you have misunderstood the word "condescending".

It is condescending to assume that the OP, Jyothiswaroop Reddy, is a fragile
and delicate little hothouse flower that needs protecting from reality where
people will tell you "Don't be so lazy and don't waste our time".

I prefer to expect more of people and let them meet my expectations, than to
downgrade my expectations and coddle them into a spiral of lower and lower
competence and ability.

(By the way Rustom, if you're reading, thank you for that link to the video a
few weeks ago about teaching 2 + 2 = 22. My blood pressure just about doubled
watching it.)




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: f-string

[Steve D'Aprano]

On Wed, 6 Dec 2017 12:21 pm, Chris Angelico wrote:

> On Wed, Dec 6, 2017 at 11:54 AM, John Pote <johnp...@jptechnical.co.uk>
> wrote:
>>
>> On 06/12/2017 00:16, Steve D'Aprano wrote:
>>>
>>> Anyone got a handy copy of Python 3.6 available to test something for me?
>>>
>>> What does compile('f"{spam} {eggs}"', '', 'single') return?

[...] 

> I think Steve just wanted to see what we'd all define spam and eggs as.
> 
> ChrisA
> *ducking for cover*

Possibly duck eggs :-)


I was thinking of responding to your comment on Python-Ideas that said there
is no way to represent an unevaluated f-string in Python, but then I decided
that even for me that was too pedantic.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: csv.DictReader line skipping should be considered a bug?

[Steve D'Aprano]

On Wed, 6 Dec 2017 11:43 am, MRAB wrote:

> A blank line could be a record if there's only one field and it's empty.


That's technically correct, but if you have only one field, its barely a CSV
file at all.

Given that CSV technically requires at least two fields (in order to have a
separator between fields) I'm not too concerned by the inability to represent
a record consisting of only a single blank field. If there was an actual CSV
standard (rather than just a collection of implementations with slightly
different behaviour) I'd be more concerned at this design lack.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: f-string

[Steve D'Aprano]

On Wed, 6 Dec 2017 11:54 am, John Pote wrote:

[...]
> Ran above test file and got,
>  >>python36 compiletest.py
>  at 0x02120E40, file "", line 1>
> 
> 
> SPAM scrambled

Thanks everyone, that's what I wanted to see.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

f-string

[Steve D'Aprano]

Anyone got a handy copy of Python 3.6 available to test something for me?

What does compile('f"{spam} {eggs}"', '', 'single') return?

What does eval()'ing the above compiled object do? If necessary, you may have
to define spam and eggs first.


Thanks in advance.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: csv.DictReader line skipping should be considered a bug?

[Steve D'Aprano]

On Wed, 6 Dec 2017 04:20 am, Jason wrote:

> I ran into this:
>
https://stackoverflow.com/questions/27707581/why-does-csv-dictreader-skip-empty-lines
> 
> # unlike the basic reader, we prefer not to return blanks,
> # because we will typically wind up with a dict full of None
> # values
> 
> while iterating over two files, which are line-by-line corresponding. The
> DictReader skipped ahead many lines breaking the line-by-line
> correspondence.

Um... this doesn't follow. If they are line-by-line corresponding, then they
should skip the same number of blank lines and read the same number of
non-blank lines.

Even if one file has blanks and the other does not, if you iterate the over
the records themselves, they should keep their correspondence.

I'm afraid that if you want to convince me this is a buggy design, you need to
demonstrate a simple pair of CSV files where the non-blank lines are
corresponding (possibly with differing numbers of blanks in between) but the
CSV readers get out of alignment somehow.


> And I want to argue that the difference of behavior should be considered a
> bug. It should be considered as such because: 1. I need to know what's in
> the file to know what class to use.

Sure. But blank lines don't tell you what class to use.

> The file content should not break at-least-1-record-per-line.

Blank lines DO break that requirement. A blank line is not a record.


> There may me multiple lines per record in the 
> case of embedded new lines, but it should never no record per line.

I disagree. A blank line is not a record. If I have (say) five fields, then:

\n

is a blank record with five empty fields. \n alone is just a blank. The
DictReader correctly returns records with blank fields.


> 2.  It's a premature optimization. If skipping blank lines is desirable,
> then have another class on top of DictReader, maybe call it
> EmptyLineSkippingDictReader.

No, that's needless ravioli code. The csv module already defines a basic
reader that doesn't skip blank lines. Having two different DictReaders, one
which doesn't work correctly because it wrongly expands blank lines to
collections of blank fields, is not helpful.

Perhaps if they were called BrokenDictReader for the one which expands blank
lines to empty records, and DictReader for the one which correctly skips
blank lines.


> 3. The intent of DictReader is to return a 
> dict, nothing more, therefore the change of behavior isn inappropriate.

No, if all you want is a dict, call dict() or use the dict display {}. The
intent of DictReader is to *read a CSV file and extract the records* as a
dict. Since blank lines aren't records, they should be skipped.


> Does anyone agree, or am I crazy?

I wouldn't want to guess your mental health based just on this isolated
incident, but if I had to make a diagnosis, I'd say, yes, crazy as a loon.

*wink*




-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Please tell me how to execute python file in Ubuntu by double clicking on file. (Posting On Python-List Prohibited)

[Steve D'Aprano]

On Tue, 5 Dec 2017 07:58 pm, Lawrence D’Oliveiro wrote:

> On Tuesday, December 5, 2017 at 3:39:26 AM UTC+13, Rick Johnson wrote:
>> 
>> Sounds like your OS file associations are all botched-up ...
> 
> Linux doesn’t do “OS file associations”.


Then how does my Linux box know that when I double-click on a text file, it
launches kwrite rather than (say) the Gimp or LibreOffice?

When I right-click on a mp4 video, I get a menu that includes a Open With
command that shows (amount others) Kaffeine, mplayer and VLC.

If you mean the Linux *kernel* doesn't do file associations, then you should
have said so.

But why do you care about the kernel? Would you think it even the *tiniest*
useful to claim that "Linux doesn't do email" because it is sendmail or
postfix (or similar) that sends email rather than the Linux kernel itself?



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why won't slicing lists raise IndexError?

[Steve D'Aprano]

On Tue, 5 Dec 2017 11:31 pm, Rick Johnson wrote:

> Ned Batchelder wrote:
> [...]
>> Your original statement sounded like, "The else clause can
>> never be executed,"
> 
> No. Of course not. Note that i mentioned _pragmatism_. My
> complaint about the else-clause was not that it could
> _never_ be executed, my complaint that was that the else-
> clause (in Terry's example) serves no useful purpose save to
> act as a semantical placeholder for a non-action.

When you find yourself in a hole, stop digging.

You've already been told that there's no indication or reason to believe that
it is a non-action. You've already been given at least one possible action.
It isn't a non-action, it is two distinct actions:

- the action you take when the slice is non-empty;

- the action you take when the slice is empty.



> Heck, it 
> doesn't even log anything! "Practicality beats purity". For
> instance:
> 
> (1) Would it be practical to make a shopping list that
> includes only the products you _don't_ want to buy?

Obviously not *only* the products you don't want, but it is very common to
specify constraints, e.g. what you don't want.

New car. Must not have an internet connected engine.

Gluten-free bread.

Unscented toilet paper.

Ingredients to make fruit salad. No bananas.

Or alternatives:

Get a slice of pizza for lunch. If they're out of pizza (i.e. the
slice is empty) get Chinese food instead.


> (2) How about a list of navigation directions that includes
> tangential info?
>   
> "After you take the second left turn, you'll see a big
> windmill -- just ignore that"

Be careful about rhetorical questions. The answer is not always what you think
it must be.

Giving directions that include landmarks is *very* useful. If you don't see
the windmill, you know you've taken the wrong turn. When you do see it, you
know you're on the right track.


> (3) How about a recipe that includes non-ingredients?

My wife is an expert at making bread-and-no-butter pudding. This should be
obvious, but in case it isn't, the reason it has that name is because it is a
modified recipe based on (can you guess what?) bread-and-butter pudding, a
British staple.

Also:

Flourless orange cake.

Taken from a recipe for cooking lentils:

"Don't add vinegar, as it makes the lentils tough."

(Apparently some people add vinegar when cooking pulses. Curious.)

And for those who are well-educated but not wise:

"The difference between being educated and being smart is, the 
educated person knows that tomato is a fruit, while the wise
person knows not to put tomato in fruit salad."

I can think of a couple of people I would need to write "No tomatoes" on any
fruit salad recipe I gave them.



>> which would have been in direct opposition to the point,
>> that slicing outside the limits would produce an empty
>> list, not an exception.
> 
> And again, that is precisely why i raised an objection to
> this else-clause. Because the inclusion of this else-clause
> is a distraction from the point.

Ah, we've entered Ricksville, where demonstrating that slicing can return an
empty list is taken as distraction from the point that slicing can return an
empty list.

No, wait, you've convinced me Rick! Your logic is impeccable! Except for one
teeny tiny flaw in your reasoning -- you have the conclusion backwards. Its
not the *empty* slice (the else clause) that is the distraction. Its the
*non* empty clause. We already know that slicing returns a list of items. The
part we're discussing is what happens when the slice is out of bounds? Does
it return an empty list, or raise an exception?


alist = bigger_list[start:finish]
if not alist:
print("Look Ma, no exception was raised!!!")


/only half serious


[...]
> item = seq[n:n+1]
> if item:
> process(item)
> else:
> do_without_item()
>  
> "do_without_item()" implies a non-action.

The mind boggles how you get by in real life, when you're so busy digging
yourself into deeper and deeper holes. Do you think about things before you
write them?

"I was going to make spaghetti and meat balls for tea tonight, but the store
was out of mince, so I'll have to do without meat balls."

Does that mean I go hungry and don't eat?


>> Perhaps I have a simulation on a conceptually infinite
>> line, but only store the actual points needed.  The "non
>> action" would be to extend the line.
> 
> That logic does not follow! How do you _extend_ an infinite
> line? :-). 

Ned said that the line stores only the points needed. If you've only used
positions between 1 and 10, only values 1 through 10 have been stored. If you
then try to access position 20, the line is extended to position 20.


> Of course, in the realm of maths, a line is by 
> definition infinite.

Not necessarily. Lines can be finite (e.g. a chord) or infinite, and if
infinite, they can be defined as extending towards infinity in both

Re: Ideas about how software should behave (was: replacing `else` with `then` in `for` and `try`)

[Steve D'Aprano]

On Wed, 8 Nov 2017 04:28 am, Ian Kelly wrote:

> Steve's manufactured interactive example ("manufactured" because
> who really uses for-else interactively?  If I really care that much
> about output formatting I'm going to put it in a script).

Me. As I have said.

I really don't appreciate you implying that I'm lying about that.

As for the question of whether anyone else uses it... that depends on whether
there is anyone else:

- using the REPL in an exploratory manner

- entering loops with more than, say, two or three lines on the fly

- that requires an additional block that has to run after the loop, without a
pause for input

- and they don't realise this until after they've started typing the loop
(it's *exploratory* coding, which means sometimes you haven't thought things
through until after you start typing)

- AND they have the insight to realise that you can use an else block to
rescue the situation without having to re-enter the lines already entered.


Given how unfamiliar for...else is, it's probably only a small number of
people that meet *all* these conditions, especially the last. I fear that
most people wouldn't have the insight to realise that you can do this --
because they either don't know for...else at all, or they have the wrong
mental model for it, or they simply aren't good at thinking outside the box.

Who knows what other "thinking outside the box" uses for for...else with no
break there are? Where there is one, there are probably others. The point is,
if you require break and make the absence a syntax error, you rule them out.

As things are today, the for block and the else block are loosely coupled.
Apart from the requirement that else must immediately follow a for (or while)
block, the language doesn't *force* there to be any coupling between the for
block and the else block. We may consider them to be separate blocks, almost
unrelated in principle (if not in practice).

That has the conceptual advantage that we can teach, learn and think about the
for and else blocks as separate concepts, which allows us to reason about
them by composition:

- we can reason about the for block as iteration over a sequence;

- if we now add an else block after it, we don't have to revise our reasoning
about the for block, we simply add the else block after it.

Which is why I was able to think outside the box and realise I could rescue my
already-typed dozen line for loop by adding an else clause.


Whereas Jon's model requires us to change our understanding of the for block:

- we reason about the for block as iteration;

- if we then add an else block, we have to go back and re-interpret the for
block as some sort of metaphorical search, whether or not it actually is a
search, before we can think about the else block. Otherwise it doesn't make
sense in his model of "search, else if not condition leading to break".

In Jon's model, if we interpret else as "else no break", then we're also left
with the mystery of what happened to the "if break" clause.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Read Firefox sqlite files with Python

[Steve D'Aprano]

On Mon, 6 Nov 2017 12:39 am, Paul Moore wrote:

> On 5 November 2017 at 01:22, Steve D'Aprano <steve+pyt...@pearwood.info>
> wrote:
>> On Sun, 5 Nov 2017 04:32 am, Steve D'Aprano wrote:
>>
>>> I'm trying to dump a Firefox IndexDB sqlite file to text using Python 3.5.
>>>
>>>
>>> import sqlite3
>>> con = sqlite3.connect('foo.sqlite')
>>> with open('dump.sql', 'w') as f:
>>> for line in con.iterdump():
>>> f.write(line + '\n')
>>
>>
>> Never mind. I dumped the file using the sqlite3 command line tool. Thank
>> you to all those who answered.
>>
>> The file contains three INSERT statements, the first two don't have
>> anything of interest, and the third (which presumably contains all the data
>> I'm trying to recover) is an opaque 600+ KB blob.
>>
>> Naturally. Why would you use a database as a database, when instead you
>> could just dump a big ball of mud into it?
> 
> Hmm, *.sql files normally contain SQL source code (as this one does).

The .sql file is the result of running .dump from the sqlite command line
tool. The original source database is 'foo.sqlite'. To be precise, it is the
database used by the Firefox Add-On "One Tab".

/home/steve/.mozilla/firefox/2z5po7dx.default/storage/permanent/indexeddb+++extension-at-one-tab-dot-com/idb/1832832054obnaet.sqlite

One Tab provides an alternative bookmark-like function, allowing you to record
URLs in groups for later use -- a bit like bookmarks. So I've been using this
for some months, until the add-on stopped working. (Yet again an automatic
update has screwed me and broken functionality.) So now I'm trying to
retrieve the bookmarks.

The database itself (the .sqlite file) is not corrupt. The sqlite CLI
processes it fine, and gives me a dump file containing a valid looking
transaction:

PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE file (id INTEGER PRIMARY KEY, refcount INTEGER NOT NULL);
CREATE TABLE "database" ( name TEXT PRIMARY KEY, origin TEXT NOT NULL, version
INTEGER NOT NULL DEFAULT 0, last_vacuum_time INTEGER NOT NULL DEFAULT 0,
last_analyze_time INTEGER NOT NULL DEFAULT 0, last_vacuum_size INTEGER NOT
NULL DEFAULT 0) WITHOUT ROWID;
...
COMMIT;


But the *interesting* part, the actual data which I hoped would be something
useful like a table of URLs, is a binary blob:

CREATE TABLE "object_data"( object_store_id INTEGER NOT NULL, key BLOB NOT
NULL, index_data_values BLOB DEFAULT NULL, file_ids TEXT, data BLOB NOT NULL,
PRIMARY KEY (object_store_id, key), FOREIGN KEY (object_store_id) REFERENCES
object_store(id) ) WITHOUT ROWID;

which then has three insertions, all of which look something like:

INSERT INTO "object_data" VALUES(1,X'307475627566',NULL,NULL,X'B8F32BA8...');

where the last argument is anything up to 11000+ hex digits. So it looks like
it might be just dumping its in-memory data structure into the database as a
blob.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Sat, 4 Nov 2017 03:57 pm, Michael Torrie wrote:

> On 11/03/2017 09:06 PM, Chris Angelico wrote:
>> On Sat, Nov 4, 2017 at 1:57 PM, Michael Torrie <torr...@gmail.com> wrote:
>>> On 11/03/2017 07:09 PM, Steve D'Aprano wrote:
>>>> On Sat, 4 Nov 2017 06:15 am, Michael Torrie wrote:
>>>>
>>>>> In fact if you have no break you may as well drop the
>>>>> else entirely, because the block will always execute.
>>>>
>>>> That's incorrect. There are multiple ways to exit a loop that will
>>>> prevent the `else` block from executing, `break` is only one.
>>>
>>> Such as?
>> 
>> There are many. But other than break, I don't know of any that WOULD
>> execute the next line of code immediately _after_ the loop.
> 
> Can you be more specific? What are some of these "many" ways of aborting
> a loop?  Help a guy out here.

Aside from more exotic methods such as os.abort, os._exit and signal handlers,
the common ways of breaking out of a loop are:

- raise
- return
- break

Am I being pedantic? Of course I am. But we're programmers -- if we don't have
an accurate and complete understanding of code, who will? Given how many
people find it difficult to understand the semanics of for...else, I think we
need to be pedantic about it.

I raise these because such commonplace ways of exiting a loop rules out
suggestions that we think of or rename `else` as "else no break"
or "finally":

(1) `else no break` is misleading as it implies that `break` is the only way
to avoid running the block;

(2) `finally` is misleading as it suggests that the block runs even when you
raise or return out of the loop, like a try...finally block.


> I know, for example, that we have exceptions. But those hardly matter in
> this discussion because they wouldn't execute the else clause either.

That contradicts what you said earlier:

"if you have no break the [else] block will always execute."

Of course you are right -- an exception will also prevent the `else` clause
from running. But that's not what you said earlier. Hence my correction.

The fact that `raise` and `return` avoid running the `else` block is not a
given. We cannot afford to say "that's so obvious it doesn't need to be
said". It isn't obvious. There are at least two other code blocks that make
guarantees about code "always"[1] running, even if you raise or return:

- in a try...except...else block, the `finally` block will still run;

- in a `with` block, the context manager's `__exit__` method will usually run
(it depends on the context manager).



[1] For reasonable values of "always". Obviously if you drop a 1000 tonne
weight on the computer, the `finally` clause will not get a chance to run
before the CPU is crushed into powder :-)


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Mon, 6 Nov 2017 01:39 am, Jon Ribbens wrote:

> On 2017-11-05, Steve D'Aprano <steve+pyt...@pearwood.info> wrote:
>> On Sat, 4 Nov 2017 04:44 am, Jon Ribbens wrote:
>>> That conforms to my model. It's searching for the condition
>>> 'count > MAX_OBJECTS'.
>>
>> That's sounds to me that you are willing to call just about any test of a
>> condition inside a loop a "search". I don't think that's helpful. I think
>> it is mangling the word to the point it is meaningless.
> 
> You're entitled to your opinion, of course. I've provided you with a
> way of thinking about 'for...else' that makes its purpose and meaning
> intuitively obvious.

By definition, if people have to learn the "right mental model" (as opposed to
the one which just comes to them naturally) then there is nothing intuitive
about it.


[...]
>> I find the code useful. I shouldn't have to justify why it is useful to me,
>> but for the record it especially comes in handy when I've already typed out
>> a multi-line loop in the REPL, and only then realised that I'll need some
>> way to add an extra line at the end.
> 
> Just press up arrow to go back and edit the first line of your
> current input and insert an 'if 1:' as someone else suggested.

"Just press up arrow" -- and that will magically fill in the other 10 lines I
typed, will it?

Don't be so condescending. I know how to use up-arrow, and I even described
using it, which you cut out of your reply.

That's a pretty dishonest trick. You saw that I had already discussed the
limitations of using up-arrow to retrieve lines from command line history,
but rather than acknowledge that, you deleted my comment and added
condescending and useless "advice" to make me seem like I don't even know
something as simple as up-arrow.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Mon, 6 Nov 2017 10:06 am, Jon Ribbens wrote:

> On 2017-11-05, Ben Finney  wrote:
>> Jon Ribbens  writes:
>>> I've provided you with a way of thinking about 'for...else' that makes
>>> its purpose and meaning intuitively obvious.
>>
>> I've read that sentence several times, and I still can't make it
>> anything but a contradiction in terms.
> 
> Well, keep at it and I'm sure you'll work it out eventually.


Alice laughed. 'There's no use trying,' she said. 'One can't 
believe impossible things.'

'I daresay you haven't had much practice,' said the Queen. 
'When I was your age, I always did it for half-an-hour a day.
Why, sometimes I've believed as many as six impossible things
before breakfast.'



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [TSBOAPOOOWTDI]using names from modules

[Steve D'Aprano]

On Mon, 6 Nov 2017 12:54 am, Stefan Ram wrote:

> Paul Moore  writes:
>>But regardless, the Zen isn't intended to be taken quite as literally
>>as the OP was trying to do. It's a statement of principles, not a set
>>of rules.
> 
>   What I am looking for is a default notation to use in my
>   beginner's tutorial and also to recommand for beginners.

When I first started learning Python, I had a bit of difficulty with the two
import forms. For about an hour, until I experimented in the interactive
interpreter until I got it.

Part of the problem at the time was that I had no concept of "namespaces" or
modules.

If your students are experienced in other languages, don't treat them like
dummies. They'll probably understand about namespaces and variable scopes.
Teach them that math.cos is the syntax for accessing a name "cos" inside
the "math" namespace, and they'll get it. Teach them that the from...import
version brings the name into the current scope, and they'll get that too. You
might have to give them analogous examples from whatever languages they're
used to.

If your students are completely new to this, like I was, then it might help to
start them with just the "import math" form, and then later introduce 
"from...import" as syntactic sugar for:

import math
cos = math.cos
del math

Although I wouldn't literally teach `del` at this point, I'd just leave it out
and and "except that the name 'math' isn't actually made available".


>   Learners have a limited capacity for learning and storing
>   information.

People don't learn isolated facts too well. But they learn *models* which they
can then infer behaviour from VERY well. This is why the Principle of Least
Surprise is so important for usable interfaces and APIs.

If you try to teach things as isolated facts, students will struggle. But:

- teach import math first;

- then show how `from math import cos` is syntactic sugar for making a local
alias for math.cos;

- then you can show that there's no particular reason why the names have to be
the same:

from math import cos as mycos

is just sugar for

import math
mycos = math.cos


But don't teach them this:

> |from math import *

There's no reason for beginners to use wildcard imports until they've mastered
*not* using wildcard imports.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Aw: Try: Except: evaluates to True every time

[Steve D'Aprano]

On Sun, 5 Nov 2017 09:53 pm, Karsten Hilbert wrote:

> On Sun, Nov 05, 2017 at 11:28:44AM +1100, Steve D'Aprano wrote:
> 
>> > Try in an interactive interpreter:
>> > 
>> >python> "a string" is True
>> 
>> Did you try that yourself?
> 
> Yes, eventually, which is why I corrected myself publicly.

Oops, didn't see that.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Zen of Python “obvious way to do it” (was: [TSBOAPOOOWTDI]using names from modules)

[Steve D'Aprano]

On Sun, 5 Nov 2017 12:49 pm, Ben Finney wrote:

> Steve D'Aprano <steve+pyt...@pearwood.info> writes:
> 
>> On Sun, 5 Nov 2017 06:42 am, Stefan Ram wrote:
>>
>> > What is the one way to do it?
>>
>> There is no philosophy of "one way to do it" in Python, that is a
>> misunderstanding (possibly deliberate...) spread about by Perl users,
>> to contrast Python from Perl's "more than one way to do it".
[...]

> I think the confusion is quite understandable, and that the Zen was
> written quite consciously referencing the (at the time quite well-known)
> Perl princple “There's more than one way to do it”.

I daresay you are right about the second part, but I'm not so sure about the
first.

Python supports both while and for loops, and recursion, and it is well known
than anything written recursively can be re-written using iteration (and vice
versa), and anything using a for-loop can be re-written using while (but not
vice versa). I don't think it is reasonable to give any credence to the idea
that Python allows "only one way" to solve problems.

Sure, it's easy to toss the phrase out without thinking, and I'm sure that in
my early years as a Python user I probably did exactly that.

I suppose that "people say things without thinking about them first" is a kind
of understanding, so I guess I'll have to admit that technically you are
right :-)


> So, I have given up trying to assign *blame* for that confusion. But
> from what I can tell it's a canard to say that the confusion is “spread
> about by Perl users”. 

Perhaps I should have said "*was*. I hardly see this "Only One Way" business
these days. A decade or two ago, it was much more common, and mostly (in my
experience) coming from Perl users as a put-down, that Python needlessly
limits what the programmer can do.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Sat, 4 Nov 2017 04:44 am, Jon Ribbens wrote:

> On 2017-11-03, Steve D'Aprano <steve+pyt...@pearwood.info> wrote:
>> The for loop does not necessarily perform a search:
>>
>> count = 1
>> for obj in sequence:
>> if count > MAX_OBJECTS:
>> print("too many objects, halting")
>> break
>> process(obj)
>> else:
>> print("finished")
>>
>> According to your mental model, this code is... what? Illegal? Silly?
>> Impossible? A syntax error?
> 
> That conforms to my model. It's searching for the condition
> 'count > MAX_OBJECTS'.

That's sounds to me that you are willing to call just about any test of a
condition inside a loop a "search". I don't think that's helpful. I think it
is mangling the word to the point it is meaningless.

How about a loop that exits at some random time? Is that a search?


for i in range(100, 0, -1):
if flip_coin() == 'Heads':
print("surprise!")
break
print("still running")
else:
print("countdown completed")


[...]
>> Your response to code that doesn't match your mental model is to say
>> that it is obviously wrong and probably buggy and should be made
>> into a syntax error if possible.
> 
> No, it isn't. Try reading again what I actually wrote.

What you actually wrote, in various posts:

You have a 'for...else' with no 'break'. Like I said, that should
probably be a syntax error.

if what the 'for' clause is doing doesn't match the concept 
of 'searching for a match' then it's obvious that you shouldn't
be using 'for...else' in the first place.

the language doesn't currently strictly enforce the requirement
for a 'break', but nevertheless if you don't have one you almost
certainly have a bug.


I stand by my comment as an accurate description of your response.


>> Ah yes, because it is inconceivable that anyone might have thought of a use
>> for for...else without a break.
> 
> It's not inconceivable, but nobody has thought of a sensible use so far
> (by which I mean one that shows it's a useful feature).

I find the code useful. I shouldn't have to justify why it is useful to me,
but for the record it especially comes in handy when I've already typed out a
multi-line loop in the REPL, and only then realised that I'll need some way
to add an extra line at the end. Since I spend a lot of time in the REPL
doing exploratory coding, this happens more often than I'd like.

Of course I could cancel the edit, then add in some sort of scaffolding code
to allow me to print an extra line at the end, then hit the up arrow
repeatedly to re-enter the lines I've already typed, editing them to fit the
scaffolding, then add the final print.

But that's tedious and painful and a waste of time and effort when it is
trivially easy to just add "else: print()". That is *simple* and it *works*
and it solves the problem.

And yes, a better REPL would also solve that problem. But I have to use the
REPL that exists, not the imaginary one that I'd like. If you want to call
this a hackish work around for a limitation of the REPL, I'll say... okay.
What's your point? It is still useful.

Try thinking outside the box. There's nothing wrong with finding unusual and
hackish uses for flow control statements.


[...]
>>> It's an incredibly obscure work-around for a different problem,
>>
>> You mean a different problem to "searching"? Yes indeed it is.
> 
> No, I mean a problem that has nothing to do with 'for...else' clauses.

I would say that it has nothing to do with the original motivation for
introducing for...else clauses into the language. Since it uses for...else,
it is pretty obtuse to deny that it has anything to do with for...else.

The for...else control statement is a tool. We can use tools for any purpose
we want, not just the ones that they were originally designed for. You are
declaring that a perfectly useful, albeit hackish, trick for the REPL is
illegitimate because it doesn't match your mental model.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Read Firefox sqlite files with Python

[Steve D'Aprano]

On Sun, 5 Nov 2017 04:32 am, Steve D'Aprano wrote:

> I'm trying to dump a Firefox IndexDB sqlite file to text using Python 3.5.
> 
> 
> import sqlite3
> con = sqlite3.connect('foo.sqlite')
> with open('dump.sql', 'w') as f:
> for line in con.iterdump():
> f.write(line + '\n')


Never mind. I dumped the file using the sqlite3 command line tool. Thank you
to all those who answered.

The file contains three INSERT statements, the first two don't have anything
of interest, and the third (which presumably contains all the data I'm trying
to recover) is an opaque 600+ KB blob.

Naturally. Why would you use a database as a database, when instead you could
just dump a big ball of mud into it?



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [TSBOAPOOOWTDI]using names from modules

[Steve D'Aprano]

On Sun, 5 Nov 2017 06:42 am, Stefan Ram wrote:

> What is the one way to do it?

There is no philosophy of "one way to do it" in Python, that is a
misunderstanding (possibly deliberate...) spread about by Perl users, to
contrast Python from Perl's "more than one way to do it".

The Zen of Python says:

There should be one-- and preferably only one --obvious way to do it.


The emphasis is on "obvious", not "one". There should be *at least* one, but
preferably only one, OBVIOUS way to solve any problem.

As for the question of importing names, the obvious way is to use a regular
import:


import math
y = math.cos(x)


which has the advantage of making it obvious where the name comes from, but
the disadvantage that it is more to type and involves an extra name lookup at
runtime, which is not free.

But:

- when performance matters

- or the name is very well known

- or you're only using a single name from the module (or at most a few)

- especially if it repeats the module name (e.g. fractions.Fraction)

it is acceptable to use the "from module import name" version:

from math import cos
y = cos(x)


Which you use depends on the situation and personal taste.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Aw: Try: Except: evaluates to True every time

[Steve D'Aprano]

On Sun, 5 Nov 2017 03:07 am, Karsten Hilbert wrote:

> Try in an interactive interpreter:
> 
>python> "a string" is True

Did you try that yourself?


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Read Firefox sqlite files with Python

[Steve D'Aprano]

I'm trying to dump a Firefox IndexDB sqlite file to text using Python 3.5.


import sqlite3
con = sqlite3.connect('foo.sqlite')
with open('dump.sql', 'w') as f:
for line in con.iterdump():
f.write(line + '\n')


The error I get is:

Traceback (most recent call last):
  File "", line 2, in 
  File "/usr/local/lib/python3.5/sqlite3/dump.py", line 30, in _iterdump
schema_res = cu.execute(q)
sqlite3.DatabaseError: file is encrypted or is not a database


If I open the file in a hex editor, it starts with:

SQLite format 3

and although I can see a few human readable words, the bulk of the file looks
like noise.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Try: Except: evaluates to True every time

[Steve D'Aprano]

On Sun, 5 Nov 2017 02:31 am, brandon wallace wrote:

> 
> I have this code that tests a server to see if it is listening on port 123
> runs and evaluates to True every time. Even if the server does not exist but
> it is not supposed to do that. I am getting no error message at all. What is
> going on with this code?

You're returning a string in both cases. Both strings evaluate as true when
treated as bools.

Success:

> return "Port 53 is reachable on: %s" % host

Failure:

> except socket.error as e:
> return "Error on connect: %s" % e
> 
> check_udp(hostname, port)


That's the first bug. The second bug is that I don't think the code does what
you think it does. You seem to be calling it with a single hostname,
presumably a string. But then you split the hostname into individual letters,
and try to connect to each of them. The *first* attempt either succeeds or
fails, and then returns.

So if you call check_udp("mailserver", 143), your function calls

for host in "mailserver":
try:
s.connect((host, port_num))


which attempts to connect to ("m", 143). That will either succeed (probably
not), or fail (probably this), and then the function returns a string, which
you apparently never look at.

I suggest you re-write your check_udp function to something more like this:


def check_udp(host, port_num):
'''Test the UDP port on a remove server.'''
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
s.connect((host, port_num))
return True
except socket.error as e:
return False


But even this is suspicious, since it is vulnerable to a "Time Of Check To
Time Of Use" bug. Just because you can connect to the host *now*, when you
call check_udp, doesn't mean it will still respond two seconds later (or even
two milliseconds later) when you attempt to connect again.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread safety issue (I think) with defaultdict

[Steve D'Aprano]

On Sat, 4 Nov 2017 05:12 am, Israel Brewster wrote:

[...]
>> People generally understand how to move data around, and the mistakes are
>> usually pretty obvious when they happen.
> 
> I think the existence of this thread indicates otherwise :-) This mistake
> was far from obvious, and clearly I didn't understand properly how to move
> data around *between processes*. Unless you are just saying I am ignorant or
> something? :-)

Yes, you were ignorant -- you didn't even realise that you were using
processes, you thought you were using threaded code when it was actually
multiprocessing code. No wonder you got it wrong.

Of course you have a good excuse: the multiprocessing is hidden deep inside
not just the library you were using, but the library *it* was using.

(I don't know how obvious the documentation of the libraries make this --
maybe they're to blame, for not being clear enough -- or maybe you were
simply ignorant about the tools you were using.)

You can't judge multiprocessing code on the basis of bugs caused by assuming
that it was threading code, writing in a threading style with shared data. If
you misuse your tools, that's not the tool's fault.

If anything, we can say that the ultimate error was that you decided to write
in a threaded style without actually using threads: the error was your
(dangerous?) choice to write non-deterministic code using shared data.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Sat, 4 Nov 2017 06:15 am, Michael Torrie wrote:

> In fact if you have no break you may as well drop the
> else entirely, because the block will always execute.

That's incorrect. There are multiple ways to exit a loop that will prevent the
`else` block from executing, `break` is only one.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Fri, 3 Nov 2017 10:49 pm, Jon Ribbens wrote:

> On 2017-11-03, Steve D'Aprano <steve+pyt...@pearwood.info> wrote:
>> On Fri, 3 Nov 2017 03:31 am, Jon Ribbens wrote:
>>> No, it's an obvious bug. You have a 'for...else' with no 'break'.
>>> Like I said, that should probably be a syntax error.
>>
>> It should absolutely not be a syntax error. There's no reason for it
>> to be a syntax error, except to satisfy some arrogant and foolish
>> idea of purity.
> 
> It'd be nice if you could be a little less rude. 

How else should we respond to the idea that something that has been legal
syntax for over twenty years and is in no way an error should be treated as a
syntax error, just to satisfy the opinion that nobody should write for...else
unless they are performing a search and it contains a break?


>> There are uncountable ways of writing code which is seemingly
>> "pointless", and we don't make it a syntax error.
> 
> And there's uncountable ways of writing code which we *do* make a
> syntax error. I'm not sure what your point is there.

None of these are syntax errors:

if False: do_something()

for x in '': pass

if True: pass

and uncountable other pieces of code which are, from some perspective,
pointless. We don't make them syntax errors because *pointless* doesn't imply
is is an error.

You think that a for...else loop with no break is pointless and "silly", even
when I demonstrate an actual use for it. Too bad. Even if it is "silly", in
your opinion, that still does not make it an error.

Is my point clear now?


>>> No, 'then' describes the opposite of what it does. The word 'then'
>>> implies something that always happens next,
>>
>> Right, which is what happens with the for...else block.
> 
> No.

Yes. It is not a matter of opinion that the `else` clause executes after
execution falls out the bottom of the loop, i.e. the loop runs, THEN the
`else` block runs.

It is an objective fact that in the absence of something which causes
execution to jump outside of the loop altogether (a return, a raise, or a
break) the execution order is loop first, then `else` block.

Both of these two sentences makes sense in English:

"Execution reaches the end of the loop, and THEN the `else` block runs."

"Execution reaches the end of the loop, or ELSE the `else` block runs."

but only the first describes what Python's for...else statement actually does.

This is not just my mental model, it is the actual objective behaviour of the
for...else statement.


> Ok, so look. It's obvious that you and I have different mental 
> models of the situation here. You're thinking of 'for...else' as two
> arbitrary clauses that run consecutively 

Scrub out the "arbitrary", and that is exactly what they are. They aren't
arbitrary: the `else` is optional, but if present it must follow after the
loop, and never the other way around.

The code inside the blocks can, of course, be any legal code we like. There's
no rule that says "only code approved by Jon Ribbens' is allowed", and
there's no rule that says "only searches are allowed".


> unless the whole thing is aborted by a 'break', 

Or return, raise, os._exit, os.abort, or having the process killed by the OS.
For simplicity we can ignore the last three (as well as more exotic
mechanisms such as "pull the power supply out" or "drop a two-tonne weight on
the computer"), but return and raise are common ways of exiting a loop and
should not be ignored or forgotten.


> whereas I'm thinking of the 'for' clause as 
> being a search for a situation that matches a condition and the
> 'else' clause being what happens if the condition is not matched
> (i.e. exactly the same as 'if...else').

That model conflates one specific use-case for the for...else statement with
the semantics of the statement, rather like insisting that "print displays a
greeting to the world" merely because 'print("Hello, World")' is so common.

The for loop does not necessarily perform a search:


count = 1
for obj in sequence:
if count > MAX_OBJECTS:
print("too many objects, halting")
break
process(obj)
else:
print("finished")


According to your mental model, this code is... what? Illegal? Silly?
Impossible? A syntax error?

Obviously wrong and almost certainly buggy?

Even when there is a search, the sense of the search may be reversed from your
mental model:


for request in sequence:
if 'please' not in request:
break
process(request)
else:
print('done')
return
raise SysExit('failure to say please')


It doesn't matter if you can think of alternative ways of writing these code
snippets. They're legal, and they work, and your mental model doesn't cover
them, at al

Re: Thread safety issue (I think) with defaultdict

[Steve D'Aprano]

On Sat, 4 Nov 2017 01:50 am, Chris Angelico wrote:

> On Fri, Nov 3, 2017 at 10:26 PM, Rhodri James  wrote:

>> I'm with Steven.  To be fair, the danger with threads is that most people
>> don't understand thread-safety, and in particular don't understand either
>> that they have a responsibility to ensure that shared data access is done
>> properly or what the cost of that is.  I've seen far too much thread-based
>> code over the years that would have been markedly less buggy and not much
>> slower if it had been written sequentially.
> 
> Yes, but what you're seeing is that *concurrent* code is more
> complicated than *sequential* code. Would the code in question have
> been less buggy if it had used multiprocessing instead of
> multithreading? 

Maybe.

There's no way to be sure unless you actually compare a threading
implementation with a processing implementation -- and they have to
be "equally good, for the style" implementations. No fair comparing the
multiprocessing equivalent of "Stooge Sort" with the threading equivalent
of "Quick Sort", and concluding that threading is better.

However, we can predict the likelihood of which will be less buggy by
reasoning in general principles. And the general principle is that shared
data tends, all else being equal, to lead to more bugs than no shared data.
The more data is shared, the more bugs, more or less.

I don't know if there are any hard scientific studies on this, but experience
and anecdote strongly suggests it is true. Programming is not yet fully
evidence-based.

For example, most of us accept "global variables considered harmful". With few
exceptions, the use of application-wide global variables to communicate
between functions is harmful and leads to problems. This isn't because of any
sort of mystical or magical malignity from global variables. It is because
the use of global variables adds coupling between otherwise distant parts of
the code, and that adds complexity, and the more complex code is, the more
likely we mere humans are to screw it up.

So, all else being equal, which is likely to have more bugs?


1. Multiprocessing code with very little coupling between processes; or

2. Threaded code with shared data and hence higher coupling between threads?


Obviously the *best* threaded code will have fewer bugs than the *worst*
multiprocessing code, but my heuristic is that, in general, the average
application using threading is likely to be more highly coupled, hence more
complicated, than the equivalent using multiprocessing.

(Async is too new, and to me, too confusing, for me to have an opinion on yet,
but I lean slightly towards the position that deterministic task-switching is
probably better than non-deterministic.)



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A use-case for for...else with no break

[Steve D'Aprano]

On Fri, 3 Nov 2017 09:13 pm, Serhiy Storchaka wrote:

> What the interpreter or configuration do you use? The standard
> interpreter uses '>>> ' as a prompt.

I have this in my Python startup file:

if (sys.version_info[0] >= 3 and os.name == 'posix'
and os.environ['TERM'] in ['xterm', 'vt100']):
# Make the prompt bold in Python 3.
sys.ps1 = '\001\x1b[1m\002py> \001\x1b[0m\002'
sys.ps2 = '\001\x1b[1m\002... \001\x1b[0m\002'
else:
sys.ps1 = 'py> '



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A use-case for for...else with no break

[Steve D'Aprano]

On Fri, 3 Nov 2017 04:22 pm, Paul Rubin wrote:

> Steve D'Aprano <steve+pyt...@pearwood.info> writes:
>> for x in something():
>> print(x, end='')
> 
> print(''.join(something()))


I hoped that people would recognise a simplified, toy example used only to
illustrate a technique, rather than an exact copy and paste of something I'm
actually doing.

Try this instead:

for s in sequence:
if isinstance(s, bytes):
try:
 s = s.decode.('utf-8')
except UnicodeDecodeError:
 s = s.decode.('latin-1')
print(s, end='')
else:
print()


Without writing a helper function, got a fancy one liner for that too?

Or another example, sometimes you really don't want to wait until the entire
sequence is processed before printing any output.

count = 0
for obj in sequence:
if count == 60:
print(' ', time.asctime())
count = 0
print('.', end='')
expensive_time_consuming_process_that_might_take_minutes_or_hours(obj)
count += 1
else:
del count
print()
print("finished at", time.asctime())


Now honestly, do you think the more complex examples illustrate the point I
was making, and the usefulness of for...else, better than the simple version?

Because I think they make them more complicated and hard to understand, and
distract from the point I am making.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread safety issue (I think) with defaultdict

[Steve D'Aprano]

On Fri, 3 Nov 2017 02:32 pm, Stefan Ram wrote:

>   Here is an excerpt from a text from Edward E. Lee:
> 
> A part of the Ptolemy Project experiment was to see
> whether effective software engineering practices could be
> developed for an academic research setting.
[...]
> No problems were observed until the code deadlocked 
> on April 26, 2004, four years later.


That is a fantastic anecdote, thank you.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread safety issue (I think) with defaultdict

[Steve D'Aprano]

On Fri, 3 Nov 2017 02:19 pm, Rustom Mody wrote:

> «The world is concurrent» [Joe Armstrong creator of Erlang]


And the world is extremely complex, complicated and hard to understand.

The point of programming is to simplify the world, not emulate it in its full
complexity.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Fri, 3 Nov 2017 03:31 am, Jon Ribbens wrote:

> On 2017-11-02, Steve D'Aprano <steve+pyt...@pearwood.info> wrote:
>> On Fri, 3 Nov 2017 12:39 am, Jon Ribbens wrote:
>>> Why would we want to make the language worse? It is fairly obvious
>>> what 'else' means,
>>
>> Yes, obvious and WRONG.
> 
> Nope, obvious and right.
>
>> for x in seq:
>> do_something()
>> else:
>> print("seq was empty")
>>
>> is an obvious, common and wrong interpretation.
> 
> No, it's an obvious bug. You have a 'for...else' with no 'break'.
> Like I said, that should probably be a syntax error.

It should absolutely not be a syntax error. There's no reason for it to be a
syntax error, except to satisfy some arrogant and foolish idea of purity.

There are uncountable ways of writing code which is seemingly "pointless", and
we don't make it a syntax error. Sometimes it is even useful. A for...else
with no break should no more be a syntax error than "if True".


>>> whereas 'then' has an obvious meaning that is in
>>> fact the opposite of what it would actually do.
>>
>> Er... is today opposite day? Because 'then' describes precisely what it
>> actually does.
> 
> No, 'then' describes the opposite of what it does. The word 'then'
> implies something that always happens next, 

Right, which is what happens with the for...else block.

The else block is executed after the for loop, unless you jump out of the
statement using return, raise or break. The flow is:

# this is what actually happens
loop
then `else` block

rather than:

# this is incorrect
loop
else (otherwise) `else` block

which implies that that `else` block runs if the loop did not.


> whereas 'else' conveys 
> the correct meaning, which is something that happens if the course
> of the preceding piece of code did not go as expected.

That's not what happens. This is WRONG:

for x in sequence:
...
else:
print("something unexpected happened")


Apart from the impossibility of deciding what "unexpected" means in general,
the behaviour is the opposite. The `else` clause executes immediately after
you fall out the bottom of the for-loop, NOT conditional on some event
(whether unexpected or not).


>>> Maybe the change should be that it is a syntax error to use a
>>> 'for/while...else' with no 'break'.
>>
>> Only if you want to make the experience of using Python in the interactive
>> interpreter worse. See my recent post:
>>
>> "A use-case for for...else with no break"
> 
> Yes, I saw that. It's possible you are the only person in the world
> ever to have done that. It would not make the interactive interpreter
> 'worse' in the slightest for that silly trick to be lost.

Just because you personally haven't used this technique doesn't make it
a "silly trick".

And while I thank you for the complement that I am the cleverest and most
insightful Python coder in the world, I'm sure I'm not. If I could think of
this technique, I daresay that many other people who are much smarter than me
have done so too.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread safety issue (I think) with defaultdict

[Steve D'Aprano]

On Fri, 3 Nov 2017 07:24 am, Chris Angelico wrote:

> On Fri, Nov 3, 2017 at 3:27 AM, Israel Brewster 
> wrote:
>>
>> Actually, that saying is about regular expressions, not threads :-) . In
>> the end, threads are as good a way as handling concurrency as any other,
>> and simpler than many. They have their drawbacks, of course, mainly in the
>> area of overhead, and of course only multiprocessing can *really* take
>> advantage of multiple cores/CPU's on a machine, but unlike regular
>> expressions, threads aren't ugly or complicated. Only the details of
>> dealing with concurrency make things complicated, and you'll have to deal
>> with that in *any* concurrency model.
>>
> 
> Thank you. I've had this argument with many people, smart people (like
> Steven), people who haven't grokked that all concurrency has costs -

Of course I grok that all concurrency has costs. Apart from comparatively rare
cases of "embarrassingly parallel" algorithms, any form of concurrent or
parallel processing is significantly harder than sequential code.


> that threads aren't magically more dangerous than other options.

There's nothing magical about it.

Threads are very much UNMAGICALLY more dangerous than other options because
they combine:

- shared data; and

- non-deterministic task switching.

Having both together is clearly more dangerous than only one or the other:

- async: shared data, but fully deterministic task switching;

- multiprocessing: non-deterministic task switching, but by default
  fully isolated data.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A use-case for for...else with no break

[Steve D'Aprano]

On Fri, 3 Nov 2017 09:20 am, Terry Reedy wrote:

> This seems like a bug in how Python interacts with your console. On 
> Windows, in Python started from an icon or in Command Prompt:
> 
>  >>> for c in 'abc': print(c, end='')
> ...
> abc>>>

That's still unfortunate: the prompt is immediately after the output, rather
than starting on a new line.


> IDLE adds \n if needed, so prompts always starts on a fresh line.
> 
>  >>> for x in 'abcdefgh':
> print(x, end='')
> 
> abcdefgh
>  >>>

The prompts and the output aren't aligned -- the prompts are indented by an
additional space. Is that intentional?

Does IDLE do this only when writing to an actual console? Because if it does
so even when output is redirected to a file, adding an extra, unexpected
newline would be a bug in IDLE.

In any case, in the interactive interpreter there are times one wishes to
follow a loop with additional code that executes immediately after the loop,
and have them do so *together* without a prompt in between. It might not even
be because you care about the output: you might be timing something, and
don't want it to pause and wait for you to type the following code.

But you cannot write this:


py> for x in seq:
... do_this()
... do_that()


as the interpreter in interactive mode requires a blank line to terminate the
indented block. But if you include such a blank:


py> for x in seq:
... do_this()
...
py> do_that()


the output of "do_this()" is separated from the output of "do_that()", which
may be inconvenient.

Another work around is:

if True:
for x in seq:
do_this()
do_that()



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Fri, 3 Nov 2017 12:39 am, Jon Ribbens wrote:

> On 2017-11-01, Alexey Muranov  wrote:
>> what do you think about the idea of replacing "`else`" with "`then`" in
>> the contexts of `for` and `try`?
>>
>> It seems clear that it should be rather "then" than "else."  Compare
>> also "try ... then ... finally" with "try ... else ... finally".
>>
>> Currently, with "else", it is almost impossible to guess the meaning
>> without looking into the documentation.
> 
> Why would we want to make the language worse? It is fairly obvious
> what 'else' means, 

Yes, obvious and WRONG.

for x in seq:
do_something()
else:
print("seq was empty")

is an obvious, common and wrong interpretation.


> whereas 'then' has an obvious meaning that is in 
> fact the opposite of what it would actually do.

Er... is today opposite day? Because 'then' describes precisely what it
actually does.

Perhaps before we continue, we should ask what you think for...else and
while...else statements do. Just to be sure we are all on the same page here.


> It seems clear that 
> 'else' is the correct word (or at least, far better than 'then').

Not clear at all. The 'for...else' block is a common source of confusion, if
it was clear what it did, people wouldn't so often get it wrong.


> Maybe the change should be that it is a syntax error to use a
> 'for/while...else' with no 'break'.

Only if you want to make the experience of using Python in the interactive
interpreter worse. See my recent post:

"A use-case for for...else with no break"



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A use-case for for...else with no break

[Steve D'Aprano]

On Thu, 2 Nov 2017 10:45 pm, Alberto Berti wrote:

>>>>>> "Steve" == Steve D'Aprano <steve+pyt...@pearwood.info> writes:
> 
> py> for x in "abcdefgh":
> Steve> ... print(x, end='')
> Steve> ...
> py> efghpy>
> 
> 
> Steve> "For ... else" to the rescue!
> 
> py> for char in "abcdefgh":
> Steve> ... print(char, end='')
> Steve> ... else:
> Steve> ... print()
> Steve> ...
> Steve> abcdefgh
> py>
> 
> else doesn't seem to bring any advantage over:
> 
> for char in "abcdefgh":
> print(char, end='')
> print()

Have you tried it in the interactive interpreter?


py> for char in "abcdefgh":
... print(char, end='')
... print()
  File "", line 3
print()
^
SyntaxError: invalid syntax




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 10:09 pm, Ben Bacarisse wrote:

> Sure, but your argument seemed to that else has entirely the wrong
> meaning (I certainly to a double take when I have to remember what it
> means) and, in that context, finally has a meaning closer to what you
> want.

That's an argument about whether "yellow" or "purple" is closer in meaning to
the word we actually want, "spicy" :-)



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 09:04 pm, Steve D'Aprano wrote:

> then (with special case) of `pass`

That should read "then except for the special case of `pass`".

Sorry.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

A use-case for for...else with no break

[Steve D'Aprano]

Occasionally it is useful to loop over a bunch of stuff in the interactive
interpreter, printing them as you go on a single line:

for x in something():
print(x, end='')

If you do that, the prompt overwrites your output, and you get a mess:


py> for x in "abcdefgh":
... print(x, end='')
...
py> efghpy>


"For ... else" to the rescue!

py> for char in "abcdefgh":
... print(char, end='')
... else:
... print()
...
abcdefgh
py> 




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 12:49 pm, Skip Montanaro wrote:

> I don't know. The word "then" doesn't connote different ways of exiting a
> loop to me ("else" doesn't really either, I will grant you that, but it's
> what we have). Here's how I would read things:
> 
>- *while* some condition holds, execute the loop, possibly breaking out,
>*then* do some finishing work
>- *for* each element in some sequence, execute the loop, possibly
>breaking out, *then* do some finishing work

And that is exactly the behaviour of the for...else and while...else
construct. You're reading it correctly.

When you say "possibly breaking out", I assume you mean breaking out of the
*entire* compound statement, because the alternative would be just silly. The
alternative would be exactly equivalent to just following the loop with some
unindented code:


while condition:
block
# on break we jump to here
finishing


In that case, it makes no sense to include a do-nothing "else" or "then"
keyword.

To put it another way, if we are trying to infer the semantics of a statement,
and can interpret it in one of two ways:

- it does something

- it does nothing

then (with special case) of `pass`, we should always assume the first case,
thus resolving the ambiguity in favour of "Guido isn't an idiot who added a
pointless keyword" :-)


> In neither case does it seem to me that you execute the finishing work only
> if you break out of the loop, but not if the loop terminates when the while
> condition becomes false or the for loop's sequence is exhausted.

If I'm reading you correctly, I think you are confused about the `else`
clause. It is incorrect to say that the `else` clause runs "only if you break
out of the loop" (as you say). The `else` clause runs *after* the loop
completes, but NOT if you jump out with break/return/raise.

If we don't jump out of the loop, the else block is equivalent to just
following the loop with unindented code:


for x in sequence:
block
else:
follows


is equivalent to:


for x in sequence:
block
follows


Hence, in the absence of any early exit from the loop, the `else` block
behaves more like an *then* rather than an "else". The behaviour of `else`
confused me utterly until I realised that the semantics of the compound
for...else was the loop to execute, then the `else` block.

So why even bother with `else`?

Unlike the second case, the `else` block is part for the compound for/while
statement. And so `break` doesn't just exit the loop, it exits the whole
compound statement, jumping past the `else`:

for x in sequence:
block
else:
follows
# break jumps to here
more code

 
The motive for allowing this pattern is described here:

https://shahriar.svbtle.com/pythons-else-clause-in-loops


> You might
> consider that while/then or for/then actually reads too much like English,
> fooling you into interpreting it as English, opening you up to ambiguity.

If you interpret it as English, it works fine. You run the loop, then you run
the "then" clause. If you jump out of the loop (whether by return, raise or
break) you jump out of the *entire* statement, not just the loop part.

There's no ambiguity because we can assume Guido wouldn't introduce a
pointless block keyword that does literally nothing except require an indent.


> You might argue that "else" doesn't either, but it has the strangely nice
> property of actually being a bit clumsier to read, forcing the reader to
> learn and apply the precise rules of the programming language instead of
> infer the more ambiguous rules of English.

If only that were true... 

There's a simple, obvious, but sadly WRONG plain English interpretation of
for...else, namely that the `else` clause runs if the for sequence was empty:


for x in L:
pass
else:
# This is wrong!
print("L is empty")


and similar for while. That lead me astray for the longest time! And I'm not
the only one.


> English and other natural languages aren't precise enough to serve as
> programming languages. 

Hmmm... I think Hypertalk might have something to say about that.

And Inform7 I think would laugh in your face :-)




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: what exactly does type.__call__ do?

[Steve D'Aprano]

On Thu, 2 Nov 2017 10:13 am, Jason Maldonis wrote:

> Hi everyone,
> 
> I want to use a metaclass to override how class instantiation works. I've
> done something analogous to using the Singleton metaclass from the Python3
> Cookbook example.

In my opinion, nine times out of ten, using a metaclass for something like
that is overkill.

(And that's putting aside the fact that 999 out of a thousand, using a
Singleton is the wrong solution, no matter what the question is.)


> However, I want to provide a classmethod that allows for "normal" class
> instantiation that prevents this metaclass from being used.

To me, that strongly suggests that a metaclass is the wrong solution.


> To do that, I think I just make a @classmethod constructor function.
> However, I can imagine a few different ways of writing this:
> 
> @classmethod
> def normal_constructor(cls, *args, **kwargs):
> return type.__call__(*args, **kwargs)

Untested, but I think that should be:

return type.__call__(cls, *args, **kwargs)


> @classmethod
> def normal_constructor(cls, *args, **kwargs):
> return super(???).__call__(*args, **kwargs)  # I'm not sure what should
> go in the super here  (I'm using python3)
> 
> @classmethod
> def normal_constructor(cls, *args, **kwargs):
> self = cls.__new__(cls)
> self.__init__(*args, **kwargs)
> return self
> 
> Is one of these correct? Or do they all do the same thing?

None of them look "correct", they all look "weird and scary" :-)

If I had to pick one of these three -- and I hope that I would not -- I'd pick
the first one.


> I was looking for documentation for what exactly `type.__call__` does so
> that I can emulate it, 

And then if type.__call__ changes, your emulation will be wrong.


> but I wasn't able to find any docs explicitly 
> detailing what that method does. If someone knows where this info is that
> would be great too.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 12:50 pm, Ben Bacarisse wrote:

> Steve D'Aprano <steve+pyt...@pearwood.info> writes:
> 
>> On Thu, 2 Nov 2017 08:12 am, Alexey Muranov wrote:
>>
>>> what do you think about the idea of replacing "`else`" with "`then`" in
>>> the contexts of `for` and `try`?
[...]
> Re-using finally would not need a new keyword and might be close enough
> in meaning.

Reusing finally would be *completely* wrong.

The semantics of `finally` is that it should be executed no matter[1] how you
exit the previous block. E.g. if we write:


try:
return 1
finally:
print("exiting")


then "exiting" is printed. Replace the return with a raise, and the same
applies. Reusing `finally` in for and while loops would imply the similar
behaviour:


for i in range(100):
return i
finally:
print("exiting")


should print "exiting", when in fact it does not. Likewise if you replace the
return with a break.





[1] Within the bounds of normal processing. There are ways to halt Python
without executing any subsequent code, namely os._exit, and os.abort dumps
core to exit immediately.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 08:23 am, Ned Batchelder wrote:

> Apart from the questions of backward compatibility etc (Python is
> unlikely to ever go through another shift like the 2/3 breakage), are
> you sure "then" is what you mean?  This won't print "end":
> 
>  for i in range(10):
>  print(i)
>  else:
>  print(end)


You are neither the first nor the last person to have mistakenly understood
the "else" clause to run only if the main loop does not.

It is a common error. I've done it. And this demonstrates exactly why the
choice of keyword is so poor. If somebody of your experience can misread it,
I don't feel so bad about how long it too me to understand it when I was a
newbie.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 08:21 am, Chris Angelico wrote:

> With the 'for' loop,
> it's a bit more arguable, but I've never seen anything more than a
> weak argument in favour of 'then'

Thhpptpt!

"else" is an completely inappropriate term that doesn't describe the semantics
of the statement even a little bit. The argument that it means "else no
break" is feeble because break is not the only way to exit the loop and
therefore skip executing the else clause.

It is not even necessarily the most common: I wouldn't be surprised if there
were more returns out of the middle of a loop than breaks, although I
wouldn't necessarily predict it either.

If we spoke in ordinary English using "else" the way Python uses it for
looping, we would say:

"Shampoo your hair twice, ELSE apply conditioner and leave for five minutes
before rinsing."

"Boil the pasta until it is soft, ELSE drain it and mix in the sauce."




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Steve D'Aprano]

On Thu, 2 Nov 2017 08:02 am, Ben Bacarisse wrote:

> r...@zedat.fu-berlin.de (Stefan Ram) writes:
> 
>> Wolfgang Maier  writes:
>>>If you're worried bout having things on separate lines, you could write:
>>>import os; os.getcwd()
>>>,etc., which is actually saving a few characters :)
>>
>>   Yes, but there still is the risk of the identifier »os«
>>   already being used in the sorrounding code. While
>>
>> __import__( "os" ).getcwd()
>>
>>   does not seem to "leak" names into the enclosing scope.
> 
> Also it's an expression which may be important in your "quick and dirty"
> scripts.

No script is so quick or so dirty to justify calling __import__ with a string
literal argument instead of import.

We've all written quick and dirty throw-away scripts where we don't care too
much about best practices. But this isn't so much less-than-best practices as
worst-practices: optimizing to save a few seconds during the initial editing
run, by using the copy-and-paste anti-pattern.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Steve D'Aprano]

On Thu, 2 Nov 2017 05:57 am, Stefan Ram wrote:

> I also have heard that there was a module cache, so I
> was hoping that a second import of the same module might
> not be such an effort for the implementation.


There is: sys.modules.

Although `import spam` is cheap when spam is in the cache, its not free, and
`__import__("spam")` is even less cheap (more costly). Its a function call,
not a statement, so it requires a runtime name lookup and a function call on
top of the same process of checking the cache and importing the module.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 08:12 am, Alexey Muranov wrote:

> Hello,
> 
> what do you think about the idea of replacing "`else`" with "`then`" in
> the contexts of `for` and `try`?


Yes, this, exactly!!!

(For while and for loops, but not try -- see below.)

I have argued this for many years. The current choice of "else" is painfully
misleading, and it causes people (including myself) to wrongly guess that
the "else" block runs only if the for/while block doesn't run at all:


# This is wrong!
for x in sequence:
...
else:
print("sequence is empty")


The actually semantics of "else" is that the block is UNCONDITIONALLY run
after the for/while loop completes, unless you jump out of the loop using
return, raise or break. That makes it a "then" block, not "else".


> It seems clear that it should be rather "then" than "else."  Compare
> also "try ... then ... finally" with "try ... else ... finally".

I disagree about the try block though. The semantics of the try block are:

try:
   A
except:
   B
else:
   C
finally:
   D

(1) code block A is attempted;

(2) IF an exception occurs, jump to code block B;

(3) otherwise (else), no exception occurs, so jump to code block C;

(4) finally run code block D on your way out, regardless of which blocks are
executed and how you exit them.


So I think "else" is correct here. The else block only gets called if there is
no exception.


> Currently, with "else", it is almost impossible to guess the meaning
> without looking into the documentation.

It is worse than that: it is easy to guess the WRONG meaning, namely that the
else block runs when the for/while loop doesn't execute at all (the for-loop
sequence is empty, or the while-loop condition is initially false).


> Off course, it should not be changed in Python 3, maybe in Python 4 or
> 5, but in Python 3 `then` could be an alias of `else` in these contexts.

Unfortunately, this is almost certainly not going to happen. It would require
adding a new keyword, and unless Guido changes his mind, he doesn't think
this change is worthwhile.

If I were the BDFL, this would be the first change I make :-)



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread safety issue (I think) with defaultdict

[Steve D'Aprano]

On Thu, 2 Nov 2017 05:53 am, Israel Brewster wrote:

[...]
> So the end result is that the thread that "updates" the dictionary, and the
> thread that initially *populates* the dictionary are actually running in
> different processes.

If they are in different processes, that would explain why the second
(non)thread sees an empty dict even after the first thread has populated it:


# from your previous post
> Length at get AC:  54 ID: 4524152200  Time: 2017-11-01 09:41:24.474788
> Length At update:  1 ID: 4524152200  Time: 2017-11-01 09:41:24.784399
> Length At update:  2 ID: 4524152200  Time: 2017-11-01 09:41:25.228853


You cannot rely on IDs being unique across different processes. Its an
unfortunately coincidence(!) that they end up with the same ID.

Or possibly there's some sort of weird side-effect or bug in Flask that, when
it shares the dict between two processes (how?) it clears the dict.

Or... have you considered the simplest option, that your update thread clears
the dict when it is first called? Since you haven't shared your code with us,
I cannot rule out a simple logic error like this:

def launch_update_thread(dict):
dict.clear()
# code to start update thread


> In fact, any given request could be in yet another 
> process, which would seem to indicate that all bets are off as to what data
> is seen.
> 
> Now that I've thought through what is really happening, I think I need to
> re-architect things a bit here. 

Indeed. I've been wondering why you are using threads at all, since there
doesn't seem to be any benefit to initialising the dict and updating it in
different thread. Now I learn that your architecture is even more complex. I
guess some of that is unavailable, due to it being a web app, but still.


> For one thing, the update thread should be 
> launched from the main process, not an arbitrary UWSGI worker. I had
> launched it from the client connection because there is no point in having
> it running if there is no one connected, but I may need to launch it from
> the __init__.py file instead. For another thing, since this dictionary will
> need to be accessed from arbitrary worker processes, I'm thinking I may need 
> to move it to some sort of external storage, such as a redis database

That sounds awful. What if the arbitrary worker decides to remove a bunch of
planes from your simulation, or insert them? There should be one and only one
way to insert or remove planes from the simulation (I **really** hope it is a
simulation).

Surely the right solution is to have the worker process request whatever
information it needs, like "the next plane", and have the main process
provide the data. Having worker processes have the ability to reach deep into
the data structures used by the main program and mess with them seems like a
good way to have mind-boggling bugs.



> Oy, I made my life complicated :-)

"Some people, when confronted with a problem, think, 'I know, I'll use
threads. Nothew y htwo probave lems."

:-)



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Steve D'Aprano]

On Thu, 2 Nov 2017 04:25 am, Stefan Ram wrote:

>   I started to collect some code snippets:
[...]

> __import__( "random" ).random()
>
>   And so on. You get the idea.
>
>   However, reportedly, all those snippets are anti-patterns
>   because they use »__import__«.

Correct. Nearly all dunder functions and methods are reserved for use by the
interpreter.

It isn't an outright error to call __import__ directly, but you should avoid
it unless absolutely necessary.


[...]
>   What I'm supposed to do instead, I guess, is:
[...]

> import random
> ...
> random.random()
> 
>   Now, the user has to cut the import, paste it to the top
>   of his code, then go back to the list of snippets, find
>   the same snippet again, copy the expression, go to his code,
>   then find the point where he wanted to insert the snippet again,
>   and finally insert the snippet.

The ellipsis do nothing, why are they there? And surely you are capable of
copying two lines at a time.


import random
random.random()


requires only one copy operation.

There is no outright requirement to collect all the imports at the top of your
module. That's merely a very good convention to follow. If you don't mind
breaking the convention, you can simply paste the result where you want the
random number:

# code here
# more code
# paste here >>
import random
random.random()  # and edit as needed


This has saved you ten seconds of editing time while writing the code. It will
probably cost you ten minutes, when you come back to maintain the program in
six months and the imports are scattered all through the module, inside
functions and classes, but that's your decision to make.


>   And still there now is a 
>   risk of name collisions. So, it seems to me that __import__
>   is just so much better!

Only if you wish to write ugly, inefficient code.

The idea of code snippets is that they are intended as *templates*, not that
you repeat them over and over again. That would be the worst sort of copy and
paste programming, an anti-pattern.

If you need four random numbers, would you write this?


a = __import__( "random" ).random()
b = __import__( "random" ).random()
c = __import__( "random" ).random()
d = __import__( "random" ).random()


Or a list of them?


numbers = [__import__( "random" ).random() for i in range(1000)]


Do I need to explain how terrible that code is?

Code snippets are not an alternative to actually thinking about your code and
writing the best code you can.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Performance of map vs starmap.

[Steve D'Aprano]

On Mon, 30 Oct 2017 09:10 pm, Kirill Balunov wrote:

> Sometime ago I asked this question at SO [1], and among the responses
> received was paragraph:
> 
>  - `zip` re-uses the returned `tuple` if it has a reference count of 1 when
> the `__next__` call is made.
>  - `map` build a new `tuple` that is passed to the mapped function every
> time a `__next__` call is made.
> 
> Why can not `map` use the same approach as `zip`?


It possibly could, if somebody could be bothered to make that
micro-optimization.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to join elements at the beginning and end of the list

[Steve D'Aprano]

On Wed, 1 Nov 2017 02:29 am, Neil Cerutti wrote:

> You can use the % operator instead of +, and a generator
> expression instead of map. It's a pretty small improvement,
> though.
> 
> values = '||%s||' % ('||'.join(str(s) for s in value_list))
> 
> At least... I THINK you can use that generator expression in 2.7.

Generator expressions are slightly slower when you call join. If join knows
how many items there are, it can allocate space more efficiently, which is
faster.

So even though it takes a bit of time to build, and throw away, a temporary
list, its actually faster to join a list comp than a generator expression.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Invoking return through a function?

[Steve D'Aprano]

On Tue, 31 Oct 2017 02:26 pm, Rustom Mody wrote:

> My own feeling about lisp-macros is conflicted:
> - They are likely the most unique feature of lisp, putting it at the top of
> the blub-language tower
> - They are the single reason Lisp can never succeed like mainstream
> languages: Any significant Lisp sub-ecosystem will inevitably develop a
> macro set which succinctly and precisely expresses its needs but is arcane
> and incomprehensible to someone from another sub-ecosystem.

Well said. That's one of the disadvantages of Forth as well: since Forth
allows you to define your own control-structures, even the structure of the
code can be unfamiliar.

Another way to put it might be that any sufficiently complex Lisp program
doesn't look like Lisp any more.

Except perhaps for the myriad parentheses *wink* 



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Invoking return through a function?

[Steve D'Aprano]

On Tue, 31 Oct 2017 02:34 pm, Chris Angelico wrote:

> On Tue, Oct 31, 2017 at 2:00 PM, Steve D'Aprano
> <steve+pyt...@pearwood.info> wrote:
>> Python has no GOTO, fortunately, but C has at least two, GOTO and LONGJMP.
>> A C macro could, if I understand correctly, jump into the middle of another
>> function. (Yay for spaghetti code!)
> 
> No, I don't think you do understand them correctly - or at least, I
> don't know of any way for a C macro to jump into the middle of a
> function.

I presume a macro could contain a call to longjmp, yes? Since longjmp can jump
into another function (albeit only one which has prepared for it in advance),
so can the macro.

https://stackoverflow.com/questions/21355110/how-to-goto-into-different-function-in-c

And what about assembly? Couldn't you jump into a function from assembly? Of
course the stack will be all wrong, but if you're using assembly you have to
manage that yourself.


> There are three quite different things mentioned here.
> 
> 1) The 'goto' statement, which unconditionally jumps you to another
> location *in the same function*

Yes. And a large enough function can contain everything. If you wanted to
write BASIC-style unstructured code, you could dump everything into one
function and use GOTO.


> 2) setjmp/longjmp, which is not actually a "goto", but more of a
> "multi-level return"

Right-oh. So sort of like an exception then.

So perhaps C is not quite as unstructured as I had initially thought.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Invoking return through a function?

[Steve D'Aprano]

On Tue, 31 Oct 2017 01:06 pm, Alberto Riva wrote:

> On 10/30/2017 10:27 AM, Rhodri James wrote:
[...]
>> You can do the same in C.  I've had the displeasure of trying to
>> maintain such code.  It was near-unreadable, because it constantly broke
>> your expectations of what the code flow *could* be.  The fact that
>> something that could return from the middle of your function without the
>> slightest indication was a rich source of bugs.
> 
> Just curious: how is this different from calling a function that throws
> an exception (that may or may not be caught by some other function
> higher up in the call stack)? That also breaks your expectations of what
> the code flow would be...

The mere possibility of an exceptions does not break the normal, unexceptional
code flow. You can still reason about the flow "if the data is valid, and
there are no exceptions" where execution proceeds from the top of the
function down, only exiting if there's a return. That's your base,
unexceptional case, and there's no traps there. Any jump (return, break,
continue) is explicit and obvious.

Then you can think about the exceptional case, where data is invalid and you
get an exception, separately. They're independent.

In the exceptional case, the control flow is different, but still predictable:
an exception is raised, and control exits the current block, to be either
caught by the nearest surrounding except block, or if no such block, halting
execution. Either way, its all relatively[1] straight-forward, with no
surprises. You can't jump into the middle of a function, you can only pop out
of the current call-chain into the surrounding function call.

But with macros, you can't do that. Anything can be anything. An ordinary
looking function call inside a loop might mask a hidden `continue` or
`break`. A function call might mask a secret `return`.

Python has no GOTO, fortunately, but C has at least two, GOTO and LONGJMP. A C
macro could, if I understand correctly, jump into the middle of another
function. (Yay for spaghetti code!)

I suppose it wouldn't be too awful if macros required dedicated syntax, so at
least you could distinguish between "this is a safe, ordinary function"
and "this is a macro, it could mean anything". But it would still increase
the maintenance burden, proportional to the number of macros used.


> To clarify, I agree with your concern. What I mean is that even in
> normal conditions there's never any guarantee that the code flow is what
> you expect it to be just by reading the body of a function.

True -- there's always *two* possible paths: the standard, unexceptional
control flow, when no exceptions are raised; and the exceptional control
flow. The second is more complex than the first, but in *practice* (if not in
theory) we can at least be reasonably sure of what exception is raised and
where it is caught.



[1] There's no doubt that exception handling, except in the simplest forms, is
more complex than code without exception handling. And this is why Rob Pike
designed Go without exceptions.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ValueError: Error 3 while encrypting in ECB mode

[Steve D'Aprano]

On Mon, 30 Oct 2017 01:20 pm, Ho Yeung Lee wrote:

 from Crypto.Cipher import AES

 key = 'mysecretpassword'
 plaintext = 'Secret Message A'
 encobj = AES.new(key, AES.MODE_ECB)
 ciphertext = encobj.encrypt("hello")
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "C:\Python27\lib\site-packages\Crypto\Cipher\_mode_ecb.py", line 124,
>   in encrypt
> raise ValueError("Error %d while encrypting in ECB mode" % result)
> ValueError: Error 3 while encrypting in ECB mode

What does the documentation for the Crypto.Cipher.AES object say?

Did you try googling for the error message?

https://duckduckgo.com/?q=Error+3+while+encrypting+in+ECB+mode

https://www.google.com.au/search?q=Error+3+while+encrypting+in+ECB+mode


which gives me this:

https://stackoverflow.com/questions/14179784/python-encrypting-with-pycrypto-aes

among other possible answers.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Invoking return through a function?

[Steve D'Aprano]

On Mon, 30 Oct 2017 06:09 am, Alberto Riva wrote:

> But that's exactly why I would like to be able to use macros. I think
> that being able to write "return if this happens" is much more explicit
> than having to write the full if statement, every time.

There have been proposals in the past for syntax similar to

return value if condition

as a short-hand for

if condition: return value

but since the only thing actually saved is a colon (or, at worst, a colon, a
newline and an indent) they went nowhere.

Sometimes the (hypothetical) gain in readability is outweighed by the increase
in unwanted language/syntactic complexity.


> The idea is that 
> you abstract a pattern giving it a name, so every time you see that name
> you know immediately what's going to happen, without having to decode
> the conditional in your mind.

Too much of a good thing... 

That's fine for when you are writing the code the first time. You see a
pattern, you give it a name, and you use that name over and over again.

But when you come back to maintain the code in five years time, you have
forgotten the name and the pattern. You see the name, not the pattern,
because you're only focusing on a small part of the code.[1] What does the
mystery name do? Unless you are very lucky and the function is very well
named[2], you have to dig back through a chain of functions and macros to
work it out.

If the pattern was small and simple enough (like the example in this
thread, "if key not in dict: return") having to dig back through its chain of
functions and macros is more costly and difficult.

Having too many named concepts to understand is as big a cognitive burden as
having too few.







[1] Software maintenance is harder, less rewarding and simply less fun than
writing code in the first place.

[2] There are only two hard problems in computer science: cache invalidation,
and naming things.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Invoking return through a function?

[Steve D'Aprano]

On Mon, 30 Oct 2017 02:35 am, Stefan Ram wrote:

>   So, I guess, we then must accept that sometimes - under
>   extraordinary circumstances - it should be tolerated to
>   write a function that is as long as six lines.

An entire six lines... you cowboy!

*wink*



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Invoking return through a function?

[Steve D'Aprano]

On Mon, 30 Oct 2017 03:35 am, Alberto Riva wrote:

> On 10/29/2017 10:35 AM, Steve D'Aprano wrote:

[...]
>> You mean *less* explicit. "checkKey" gives absolutely no hint that it
>> causes the current function to return.
> 
> That's just because I used a name that was too generic in my example. I
> can call it "returnIfKeyMissing", and then it would be clear.

No it wouldn't. That sounds like it is the "returnIfKeyMissing" function that
returns if the key is missing, and presumably pauses forever, never
returning, if the key exists.

Of course we can *learn* that this is not the case, by reading the docs (if it
has any) or the source code (if it is available). But what of every other
function, any one of which might have been named like your first example:

checkKey

We cannot rely on naming conventions, because people won't always follow it.
*Any* function could potentially return early from the caller.

But honestly, if you're prepared to write:

returnIfKeyMissing(dict, key)

that's LONGER than:

if key not in dict: return

Since your motive appears to be the wish to save typing at the expense of
readability and maintainability, then I expect that *in practice* you
wouldn't use "returnIfKeyMissing" but would prefer a less explicit, shorter
name like "checkKey".


[...]
>> You really should re-think your strategy. Your suggestion, if possible,
>> would lead to difficult to maintain code where you couldn't easily tell
>> where the exit points of a function where.
> 
> But again, it's just a naming problem. 

It *really isn't* a naming problem. Being able to force the caller to return
is a kind of GOTO, and it violates both the letter and the spirit of the
principle that functions should have "one entry, one exit".

Of course we already violate the second part of that, by allowing multiple
returns. But that doesn't mean that any other violation should be acceptable.

Allowing functions to force their caller to exit allows them to violate the
caller's post-condition contracts.

But even if it were just a naming problem, you under-estimate its magnitude.
The word "just" is not justified here: even treated as a naming problem, it
is a big problem. Not an end-of-the-world problem, but still big enough.


> Something like returnIfKeyMissing 
> would make it easy to tell where the exit points are.

No it wouldn't, because not everyone is going to use such a clumsy, long
naming convention, either out of ignorance or for some other reason. Maybe
you're using a library where everything is named in Dutch, or Russian. Maybe
the function was named "foo()" long ago, and has only subsequently gained the
ability to force the caller to return but the name was never changed.


> And as Bartc 
> pointed out, we already have this situation with exceptions, so it would
> be nothing new.

Pardon me, but it was *me* who pointed out the analogy with exceptions, not
Bart. But this is different from an exception.

The similarity is quite weak:

- any function can raise an exception;

- your hoped for feature would allow any function to cause the 
  caller to return;

But the differences are profound:

- exceptions are an alternative to normal execution flow, they don't
  silently continue unless explicitly caught and silenced;

- if you don't explicitly catch the exception, it is obvious when 
  one occurs, because your program halts and you get an obvious 
  stacktrace;

- your suggested force-caller-to-return would silently alter the
  control flow of the caller, without warning or signal that it
  had happened.

That's a very significant difference.



> Indeed, what I'm asking for could be accomplished by 
> wrapping the body of each function in a try/catch block for an ad-hoc
> exception type.

A terrible idea. Why not just write a "validate input" function that checks
your input and returns True for valid and False for invalid, then:

if not valid(args):
return

That's little harder to type than

returnIfNotValid(args)

and much more comprehensible.


> But again, since the language doesn't have macros

And this is the sort of misfeature why Guido is dead-set against Python ever
getting them.

Yes, we get it that Lisp macros are really, really powerful. But there are two
bottle-necks in coding:

- writing the code in the first place;

- maintaining the code afterwards.

Macros are aimed at simplifying the first, at the cost of the second. That's
not a strategy Python follows.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Invoking return through a function?

[Steve D'Aprano]

On Mon, 30 Oct 2017 01:18 am, Alberto Riva wrote:

> Hello,
> 
> I'm wondering if there is a way of writing a function that causes a
> return from the function that called it. To explain with an example,
> let's say that I want to exit my function if a dict does not contain a
> given key. I could write:
> 
> def testFun():
>...
>if key not in dict:
>  return
>...
> 
> But if this is a test I need to do a lot of times, I'd like to replace
> it with something shorter and more explicit:
>
> def testFun():
>...
>checkKey(dict, key)
>...

You mean *less* explicit. "checkKey" gives absolutely no hint that it causes
the current function to return.


> and I'd like checkKey to cause a return *from testFun*. In a language
> like Lisp this would be accomplished by defining checkKey as a macro
> that expands into the code shown in my first example, so that the return
> would be inside testFun and not insted checkKey. Is there a way of doing
> something like this in Python?

Fortunately not.


> Another way of phrasing my question is: is there a way to cause a return
> from a function that is higher up in the call stack, rather than the
> currently active one, without using try/except?

No.

You really should re-think your strategy. Your suggestion, if possible, would
lead to difficult to maintain code where you couldn't easily tell where the
exit points of a function where. Imagine reading code like:


def foo(x):
a = sin(x)
b = cos(x)
print(a, b)
return a + b

There's one return, right? No. If Python could do what you are asking for,
*every function call* could be a hidden, secret return. What a nightmare that
would be.

It is bad enough that any function could raise an exception, but at least
exceptions halt the normal execution of code (unless explicitly caught). They
don't silently continue normal execution.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Compression of random binary data

[Steve D'Aprano]

On Sun, 29 Oct 2017 01:56 pm, Stefan Ram wrote:

>   If the entropy of an individual message is not defined,
>   than it is still available to be defined. I define it
>   to be log2(1/p), where p is the probability of this
>   message. I also choose a unit for it, which I call "bit".

That is exactly the definition of self-information:

https://en.wikipedia.org/wiki/Self-information

See also:

https://en.wikipedia.org/wiki/Entropy_(information_theory)

which lists several forms of related measures of information:

- the self-information of an individual message or symbol taken from a 
  given probability distribution;

- the entropy of a given probability distribution of messages or symbols;

- the entropy rate of a stochastic process.


It also suggests a connection between information entropy and thermodynamic
entropy, namely that the information entropy of a system is the amount of
additional information needed to determine the microstate of a system (the
states of all its particles), given the macrostate (identified by the bulk
thermodynamic parameters such as temperature, volume, energy).

More here: 

https://physics.stackexchange.com/questions/263197/is-information-entropy-the-same-as-thermodynamic-entropy




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Compression of random binary data

[Steve D'Aprano]

On Sun, 29 Oct 2017 06:03 pm, Chris Angelico wrote:

> On Sun, Oct 29, 2017 at 6:00 PM, Ian Kelly  wrote:
>> On Oct 28, 2017 5:53 PM, "Chris Angelico"  wrote:
>>> One bit. It might send the message, or it might NOT send the message.
>>
>> Not sending the message is equivalent to having a second possible message.
> 
> Okay, now we're getting seriously existential. Is a non-message a message?

Is zero a number?

Is bald a hair colour?

Is atheism a religion?

Aristotle seriously argued that the smallest whole number was two. Zero was
clearly nothing at all, and one wasn't a *number*, it was *unity*. A number
is, by definition, a multitude of units.

https://philosophy.stackexchange.com/questions/19533/why-does-aristotle-suggest-one-is-not-a-number

http://classics.mit.edu/Aristotle/metaphysics.14.xiv.html

No message can be a message. British nuclear submarines during the Cold War
had orders that if they failed to receive any transmissions from London
within a certain time period, they were to crack open the secret orders from
the Prime Minister. Nobody knows what is in the orders, as they were
destroyed when the PM left office, but they are popularly supposed to have
included:

- fire your missiles at Russia;

- surrender to whoever won the war;

- try to flee to Australia or New Zealand and join up with whatever remnants
  of the British Empire still exist;

- do whatever you want.

It makes a good, but ultimately futile, exercise to try to guess what the
various PMs would have said.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Compression of random binary data

[Steve D'Aprano]

On Sun, 29 Oct 2017 02:31 pm, Gregory Ewing wrote:

> Steve D'Aprano wrote:
>> I don't think that's right. The entropy of a single message is a
>> well-defined quantity, formally called the self-information.
>> 
>> https://en.wikipedia.org/wiki/Self-information
> 
> True, but it still depends on knowing (or assuming) the
> probability of getting that particular message out of
> the set of all possible messages.

Indeed.


> This is *not* what danceswithnumbers did when he
> calculated the "entropy" of his example bit sequences.
> He didn't define the set they were drawn from or
> what their probabilities were.

I'm not defending or supporting Danceswithnumbers in his confusion. I'm just
pointing out that an entropy-like measure of information for individual
messages does exist, and people do often call it "entropy".



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Compression of random binary data

[Steve D'Aprano]

On Sun, 29 Oct 2017 07:03 am, Peter Pearson wrote:

> On Thu, 26 Oct 2017 19:26:11 -0600, Ian Kelly  wrote:
>>
>> . . . Shannon entropy is correctly calculated for a data source,
>> not an individual message . . .
> 
> Thank you; I was about to make the same observation.  When
> people talk about the entropy of a particular message, you
> can bet they're headed for confusion.

I don't think that's right. The entropy of a single message is a well-defined
quantity, formally called the self-information. The entropy of a data source
is the expected value of the self-information of all possible messages coming
from that data source.

https://en.wikipedia.org/wiki/Self-information

We can consider the entropy of a data source as analogous to the population
mean, and the entropy of a single message as the sample mean. A single
message is like a sample taken from the set of all possible messages.

Self-information is sometimes also called "surprisal" as it is a measure of
how surprising an event is. If your data source is a coin toss, then actually
receiving a Heads has a self-information ("entropy") of 1 bit. That's not
very surprising. If your data source is a pair of fair, independent dice,
then the self-information of receiving a two and a four is 5.170 bits. Its a
logarithmic scale, not linear: if the probability of a message or event is p,
the self-information of that event is log_2 (1/p) bits.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Compression of random binary data

[Steve D'Aprano]

On Fri, 27 Oct 2017 09:53 am, Ben Bacarisse wrote:

> A source of random can be defined but "random data" is much more
> illusive.

Random data = any set of data generated by "a source of random".





-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list