Re: Python Worst Practices
On Sat, 28 Feb 2015 17:36:44 +1100, Steven D'Aprano wrote:
> Dan Sommers wrote:
>> And thank goodness for that! I've been writing Python code since
>> 1997 and version 1.5.,¹ and I still do a double take when
>> emacs colors all my ids that faint blue that means "builtin."
> Although it is not helpful for people using screen-readers, and may be
> of limited use to the colour-blind, I am in favour of colourising
> built-ins so they stand out.
Now if only emacs were clever enough *not* to colorize "id" when it's
one of my names and not the builtin... ;-)
> On the other hand, I recall seeing an editor which rejected the idea
> of colour-coding built-ins, keywords etc., instead it coloured your
> own variables. So given:
>
> spam = 23
> eggs += cheese*len(sausage)
>
> spam, eggs, cheese and sausage would be different colours. The idea
> being, when scanning a large code base, all the places that use a
> specific variable would stand out ("just look for the dark green
> word").
As a mostly visual person, I can see (pun intented) the logic and the
value in that. I wonder how many variables could be easily
distinguished, though, before running out of easily distinguishable
colors. Then again, a clever underlying algorithm might choose colors
based on *dissimilarity* of the identifiers, so that "i" and "j" would
be very diffent colors, but "spam" and "throat_warbler_mangrove" could
be the same color because they look so different anyway.
>> I don't think I've ever used the builtin function id in a program.
>> Ever. Not even once. Honestly, what is a valid use case?
>
> Here's one. I think it's the only time I have seen id() used apart from
> interactive experimentation:
>
> https://code.activestate.com/recipes/577504
Hah. Very nice. That sort of thing is probably useful for detecting
self-referential objects, too (e.g., to prevent infinite output for a
circular list).
>> ¹ No, not continuously. I have eaten and slept since then.
>
> Slacker!
Sorry. I'll make up the hours later, I promise!
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On 02/27/2015 10:36 PM, Steven D'Aprano wrote: > Dan Sommers wrote: > >> On Sat, 28 Feb 2015 12:09:31 +1100, Steven D'Aprano wrote: >> >>> There's no harm in calling a local variable "id", if you don't use the >>> built-in id() inside that function. That's one of the reasons why >>> functions exist, so that the names you use inside a function are distinct >>> from those outside. >> >> And thank goodness for that! I've been writing Python code since 1997 >> and version 1.5.,¹ and I still do a double take when emacs >> colors all my ids that faint blue that means "builtin." > > Although it is not helpful for people using screen-readers, and may be of > limited use to the colour-blind, I am in favour of colourising built-ins so > they stand out. Sure, for the ones I use as built-ins. But I went through the color file for vim and took out the built-ins I use regularly as variables -- and 'id' was the first one to go. -- ~Ethan~ signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
Dan Sommers wrote:
> On Sat, 28 Feb 2015 12:09:31 +1100, Steven D'Aprano wrote:
>
>> There's no harm in calling a local variable "id", if you don't use the
>> built-in id() inside that function. That's one of the reasons why
>> functions exist, so that the names you use inside a function are distinct
>> from those outside.
>
> And thank goodness for that! I've been writing Python code since 1997
> and version 1.5.,¹ and I still do a double take when emacs
> colors all my ids that faint blue that means "builtin."
Although it is not helpful for people using screen-readers, and may be of
limited use to the colour-blind, I am in favour of colourising built-ins so
they stand out.
On the other hand, I recall seeing an editor which rejected the idea of
colour-coding built-ins, keywords etc., instead it coloured your own
variables. So given:
spam = 23
eggs += cheese*len(sausage)
spam, eggs, cheese and sausage would be different colours. The idea being,
when scanning a large code base, all the places that use a specific
variable would stand out ("just look for the dark green word").
> I don't think I've ever used the builtin function id in a program.
> Ever. Not even once. Honestly, what is a valid use case?
Here's one. I think it's the only time I have seen id() used apart from
interactive experimentation:
https://code.activestate.com/recipes/577504
> That said, I
> do have boatloads of parameters and objects locally named id because
> it's idiomatic (at least to me) and mnemonic (at least to me) and just
> as meaningful.
>
> ¹ No, not continuously. I have eaten and slept since then.
Slacker!
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On 2015-02-28 12:09, Steven D'Aprano wrote: > > * Make your language have a lot of keywords. Enough to make > > memorizing them ALL unlikely, requiring constant visits to your > > documentation > > Is 33 a lot? > > py> import keyword > py> keyword.kwlist > ['False', 'None', 'True', 'and', 'as', 'assert', 'break', 'class', > 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', > 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', > 'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', > 'with', 'yield'] A quick google-and-tally for languages and their corresponding number of keywords: C: 33 C#: 77 C++: 86 Java: 50 Lua: 21 PHP: 67 Pascal: 54 Perl: 40 Pike: 37 (Just for you, ChrisA) Python: 31 (2.x) or 33 (3.x) Ruby: 40 So I can't say that Python's all that bad in comparison to most other mainstream languages, with only the austere Lua beating out Python. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On Sat, 28 Feb 2015 12:09:31 +1100, Steven D'Aprano wrote: > There's no harm in calling a local variable "id", if you don't use the > built-in id() inside that function. That's one of the reasons why functions > exist, so that the names you use inside a function are distinct from those > outside. And thank goodness for that! I've been writing Python code since 1997 and version 1.5.,¹ and I still do a double take when emacs colors all my ids that faint blue that means "builtin." I don't think I've ever used the builtin function id in a program. Ever. Not even once. Honestly, what is a valid use case? That said, I do have boatloads of parameters and objects locally named id because it's idiomatic (at least to me) and mnemonic (at least to me) and just as meaningful. ¹ No, not continuously. I have eaten and slept since then. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On Sat, Feb 28, 2015 at 12:32 PM, wrote: > For example, I've seen someone create a Socket class, then created an > operator overload that allowed you to "add" a string to your socket to make > the socket send the string, with the result being a status code indicating > success or an error. > Why not left shift the socket by that string, the result being the original socket? At least that has precedent... ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On Friday, February 27, 2015 at 5:09:49 PM UTC-8, Steven D'Aprano wrote:
> Travis Griggs wrote:
>
> > If I were giving a talk at SPLASH (or some other suitable polyglot
> > conference), I might do one called "Language Design Worst Practices".
> >
> > One of my first slides might be titled:
> >
> > Abuse Common Tokens in Confusing Ways
> >
> > * Make your language have a lot of keywords. Enough to make memorizing
> > them ALL unlikely, requiring constant visits to your documentation
>
> Is 33 a lot?
>
> py> import keyword
> py> keyword.kwlist
> ['False', 'None', 'True', 'and', 'as', 'assert', 'break', 'class',
> 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for',
> 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal',
> 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']
>
>
> > * Make sure said keywords are many of the obvious words programmers would
> > use in their applications (map, object, bytes, dir, etc)
>
> Luckily, Python doesn't make that mistake of making built-ins keywords. That
> would require making actual changes to the parser each time a new built-in
> function was added, as well as breaking people's existing code.
>
> Fortunately, Python has a much better system: a small set of keywords, very
> few of which would make useful variable names ("else = 23"), and a much
> larger set of regular names in a built-in namespace.
>
>
> py> import builtins # use __builtin__ in Python 2
> py> sorted(vars(builtins).keys())
> ['ArithmeticError', 'AssertionError', ... 'type', 'vars', 'zip']
>
>
> There's 147 of the built-ins in Python 3.3, although a few of those aren't
> truly built-in, merely added at interpreter startup.
>
> The ability to shadow built-ins is not a bug, it is a feature. It's an
> amazingly powerful feature, and not particularly newbie-friendly, but
> *many* things are not easy for newbies to master or avoid abusing.
>
> - Code can override, or overload, built-ins, either at the level of
> an entire module, or inside a particular function.
>
> - Modules can offer functions which clash with a built-in name.
> E.g. reprlib.repr, math.pow.
>
> - More importantly, modules can offer stable APIs with no fear that
> the introduction of a new built-in function will require them to
> change their function's name.
>
> - Which is a special case of a more general benefit, the introduction
> of a new built-in name does *not* break existing code that already
> uses that name.
>
>
> Newbies misuse this feature because they still have a wishful-thinking
> approach to programming. One example of wishful-thinking is the common
> newbie mistake of wondering why their loop variable never changes:
>
> # Toss a coin until you get Tails.
> x = random.random()
> while x < 0.5:
> print "Heads"
> print "Tails"
>
> Isn't it obvious that I want x to get a new random number every time through
> the loop? I wish the computer understood me so I didn't need to write all
> the steps out.
>
>
> Likewise:
>
> int = 23
> n = int("42")
>
> Isn't it obvious that the second use of int has to be the built-in function?
> I wish that the computer would understand from context which one I mean.
>
> Other newbie stylistic mistakes which can increase the chance of shadowing
> errors include:
>
> * Too many overly generic variable names like "int" and "str".
>
> * Insufficient use of functions and too much top-level code. When they
> shadow a built-in, they shadow it everywhere.
>
> * Excessively large functions that do too much. By the time they reach
> the end of their 300 line function, they have forgotten that they
> have already used "list" for a variable name.
>
>
> However, even experienced developers can make this mistake too. Generally
> speaking, it's trivially easy to recover from. Although if you're doing it
> *regularly* that might be a hint of deeper problems, e.g. poor variable
> naming skills, too much top-level code.
>
> There's no harm in calling a local variable "id", if you don't use the
> built-in id() inside that function. That's one of the reasons why functions
> exist, so that the names you use inside a function are distinct from those
> outside.
>
>
>
> > * Design your syntax so that you can't disambiguate them contextually
> > between bind and reference
>
> Do you have an example of where Python cannot distinguish between a binding
> operation and a reference?
>
>
> > * Be sure to use it in a late bound language where no warnings will be
> > provided about the mistake you're making at authorship time, deferring the
> > educational experience to sundry run times
>
> Python raises a SyntaxError at compile time, not run time, if you try to
> bind to a keyword:
>
> py> global = 23
> File "", line 1
> global = 23
>^
> SyntaxError: invalid syntax
>
>
>
>
> --
> Steven
Very well-said!
Just because a feature (In this case, shadowing built-in functions) can be
ab
Re: Python Worst Practices
Travis Griggs wrote:
> If I were giving a talk at SPLASH (or some other suitable polyglot
> conference), I might do one called “Language Design Worst Practices”.
>
> One of my first slides might be titled:
>
> Abuse Common Tokens in Confusing Ways
>
> * Make your language have a lot of keywords. Enough to make memorizing
> them ALL unlikely, requiring constant visits to your documentation
Is 33 a lot?
py> import keyword
py> keyword.kwlist
['False', 'None', 'True', 'and', 'as', 'assert', 'break', 'class',
'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for',
'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal',
'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']
> * Make sure said keywords are many of the obvious words programmers would
> use in their applications (map, object, bytes, dir, etc)
Luckily, Python doesn't make that mistake of making built-ins keywords. That
would require making actual changes to the parser each time a new built-in
function was added, as well as breaking people's existing code.
Fortunately, Python has a much better system: a small set of keywords, very
few of which would make useful variable names ("else = 23"), and a much
larger set of regular names in a built-in namespace.
py> import builtins # use __builtin__ in Python 2
py> sorted(vars(builtins).keys())
['ArithmeticError', 'AssertionError', ... 'type', 'vars', 'zip']
There's 147 of the built-ins in Python 3.3, although a few of those aren't
truly built-in, merely added at interpreter startup.
The ability to shadow built-ins is not a bug, it is a feature. It's an
amazingly powerful feature, and not particularly newbie-friendly, but
*many* things are not easy for newbies to master or avoid abusing.
- Code can override, or overload, built-ins, either at the level of
an entire module, or inside a particular function.
- Modules can offer functions which clash with a built-in name.
E.g. reprlib.repr, math.pow.
- More importantly, modules can offer stable APIs with no fear that
the introduction of a new built-in function will require them to
change their function's name.
- Which is a special case of a more general benefit, the introduction
of a new built-in name does *not* break existing code that already
uses that name.
Newbies misuse this feature because they still have a wishful-thinking
approach to programming. One example of wishful-thinking is the common
newbie mistake of wondering why their loop variable never changes:
# Toss a coin until you get Tails.
x = random.random()
while x < 0.5:
print "Heads"
print "Tails"
Isn't it obvious that I want x to get a new random number every time through
the loop? I wish the computer understood me so I didn't need to write all
the steps out.
Likewise:
int = 23
n = int("42")
Isn't it obvious that the second use of int has to be the built-in function?
I wish that the computer would understand from context which one I mean.
Other newbie stylistic mistakes which can increase the chance of shadowing
errors include:
* Too many overly generic variable names like "int" and "str".
* Insufficient use of functions and too much top-level code. When they
shadow a built-in, they shadow it everywhere.
* Excessively large functions that do too much. By the time they reach
the end of their 300 line function, they have forgotten that they
have already used "list" for a variable name.
However, even experienced developers can make this mistake too. Generally
speaking, it's trivially easy to recover from. Although if you're doing it
*regularly* that might be a hint of deeper problems, e.g. poor variable
naming skills, too much top-level code.
There's no harm in calling a local variable "id", if you don't use the
built-in id() inside that function. That's one of the reasons why functions
exist, so that the names you use inside a function are distinct from those
outside.
> * Design your syntax so that you can’t disambiguate them contextually
> between bind and reference
Do you have an example of where Python cannot distinguish between a binding
operation and a reference?
> * Be sure to use it in a late bound language where no warnings will be
> provided about the mistake you’re making at authorship time, deferring the
> educational experience to sundry run times
Python raises a SyntaxError at compile time, not run time, if you try to
bind to a keyword:
py> global = 23
File "", line 1
global = 23
^
SyntaxError: invalid syntax
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: requesting you all to please guide me , which tutorials is best to learn redis database
> i want to learn redis database and its use via python , please guide me > which tutorials i should be study, so that i can learn it in good way How about https://pypi.python.org/pypi/redis/? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On Fri, Feb 27, 2015 at 2:21 PM, Travis Griggs wrote: > * Make your language have a lot of keywords. Enough to make memorizing them > ALL unlikely, requiring constant visits to your documentation > * Make sure said keywords are many of the obvious words programmers would use > in their applications (map, object, bytes, dir, etc) None of those are keywords. Keywords are these: https://docs.python.org/3/reference/lexical_analysis.html#keywords > * Design your syntax so that you can’t disambiguate them contextually between > bind and reference Maybe I misunderstand your complaint, but Python draws a sharp syntactic distinction between references and assignment targets: the latter are only ever found to the left of an = in an assignment statement; the former are never found there. There is no reason why an editor should be unable to tell the difference. > * Be sure to use it in a late bound language where no warnings will be > provided about the mistake you’re making at authorship time, deferring the > educational experience to sundry run times You should lint your code to get warnings about this (and many other things) at authorship time. A good editor should also provide some visual warning when a built-in is shadowed. -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On Sat, 28 Feb 2015 04:45:04 +1100, Chris Angelico wrote: > Perhaps, but on the other hand, the skill of squeezing code into less > memory is being replaced by other skills. We can write code that takes > the simple/dumb approach, let it use an entire megabyte of memory, and > not care about the cost... and we can write that in an hour, instead of > spending a week fiddling with it. Reducing the development cycle time > means we can add all sorts of cool features to a program, all while the > original end user is still excited about it. (Of course, a comparison > between today's World Wide Web and that of the 1990s suggests that these > cool features aren't necessarily beneficial, but still, we have the > option of foregoing austerity.) > > > ChrisA again I am fluid on this 'Clever' programming is often counter productive & unmaintainable, but the again lazy programming can also be just as bad fro this, there is no "one size fits all" solution but the modern environment does make lazy programming very easy. -- After all, all he did was string together a lot of old, well-known quotations. -- H.L. Mencken, on Shakespeare -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On Fri, 27 Feb 2015 19:14:00 +, MRAB wrote: >> > I suppose you could load the basic parts first so that the user can > start working, and then load the additional features in the background. > quite possible my opinion on this is very fluid it may work for some applications, it probably wouldn't for others. with python it is generally considered good practice to import all modules at the start of a program but there are valid cases fro only importing a module if actually needed. -- Some people have parts that are so private they themselves have no knowledge of them. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On 27 February 2015 20:06:25 GMT+00:00, Simon Ward wrote: > >I mentioned the true and false. OK, so it's a meme, but it's based on a >false (pun intended) understanding of exit status codes. That success >evaluates to true and failure evaluates to false does not mean the >values of truth and falseness are inverted. No programming language >other than that provided by system shells I have used evaluates 0 to >true. > >The shell case is unique because it typically runs processes and has to >deal with exit status codes. It feels quite natural to me that, if >COMMAND completes successfully (exit code 0) then writing "if COMMAND ; >then : true bit ; else : false bit ; fi" should execute the true bit, >but that doesn't mean all boolean logic is inverted. I pointed my partner at this thread, and she said we were using too many of the same words to mean different things. To quote: "you're both using yes or no terms to answer a question that doesn't necessarily need a "yes" or "no" answer. It's still a point of discussion (existentialism next!) but the basic idea seems to be that once the exit code is understood as an error value it is not necessarily true or false. This is what I tried to say originally so I don't disagree. My partner is not a software developer by profession, she deals with food and nutrition and bakes nice cakes. I have encouraged her to learn Python to help with some of her work, which involves calculation of nutritional values and a lot of statistics. Programming idioms should not be lost on her, but she can, and does, point out the bits she is unfamiliar with our has trouble understanding. Simon -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On 02/27/2015 04:40 PM, Chris Angelico wrote: On Sat, Feb 28, 2015 at 8:37 AM, Dave Angel wrote: Right. In C and C++, instead of being the first slide, it'd be the first 3 or 4. Between header file conflicts (especially good because the stdlib itself has many multiply-defined symbols, duplicate header files, and contradictory include path patterns) Yeah, Python has some issues with sys.path and how your local module can unexpectedly shadow a stdlib one, but at least the stdlib itself doesn't have any conflicts. I should not ever have to do this dance: #include #undef SOME_SYMBOL #include But sadly, I do I can remember resorting to #pragmas to print out clues to include order, when recursive includes were masking what on earth was going wrong. Imagine, compile time printing. Fortunately, in Forth and Python, there's much less distinction between compile time and run time. And in Python, lots of metadata and reflection data is left lying around to be inspected in case of confusion. Don't get me wrong, I love C++, even if I don't use it the way the purists would like. But there are many warts that drive me up the wall, and consequently I avoid many features which could improve my C++ life. I haven't found nearly as many in Python. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Future of Pypy?
Steven D'Aprano writes: > An interesting point of view: threading is harmful because it removes > determinism from your program. > http://radar.oreilly.com/2007/01/threads-considered-harmful.html Concurrent programs are inherently nondeterministic because they respond to i/o events that can happen in any order. I looked at the paper cited in that article and it seemed like handwaving. Then it talks about threaded programs being equivalent if they are the same over all interleavings of input, and then goes on about that being horribly difficult to establish. It talked about program inputs as infinite sequences of bits. OK, a standard conceit in mathematical logic is to call an infinite sequence of bits a "real number". So it seems to me that such a proof would just be a theorem about real numbers or sets of real numbers, and freshman calculus classes are already full of proofs like that. The presence of large sets doesn't necessarily make math all that much harder. The test suite for HOL Light actually uses an inaccessible cardinal, if that means anything to you. IOW he says it's difficult and maybe it is, but he doesn't make any attempt to explain why it's difficult, at least once there's some tools (synchronization primitives etc.) to control the concurrency. He seems instead to ignore decades of work going back to Dijkstra and Wirth and those guys. It would be a lot more convincing if he addressed that existing literature and said why it wasn't good enough to help write real programs that work. He then advocates something he calls the "PN model" (processes communicating by message passing) but that seems about the same as what I've heard called communicating sequential processes (CSP), which are the execution model of Erlang and is what I've been using with Python threads and queues. Maybe there's some subtle difference. Anyway there's again plenty of theory about CSP, which are modelled with Pi-calculus (process calculus) which can be interpreted in lambda calculus, so sequential verification techniques are still useful on it. Hmm, I see there's a Wikipedia article "Kahn process networks" about PN networks as mentioned, so I guess I'll look at it. I see it claims a KPN is deterministic on its inputs, while I think CSP's might not be. > Some discussion of the pros and cons of threading: > http://c2.com/cgi/wiki?ThreadsConsideredHarmful This wasn't very informative either. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On Sat, Feb 28, 2015 at 8:37 AM, Dave Angel wrote: > Right. In C and C++, instead of being the first slide, it'd be the first 3 > or 4. Between header file conflicts (especially good because the stdlib > itself has many multiply-defined symbols, duplicate header files, and > contradictory include path patterns) Yeah, Python has some issues with sys.path and how your local module can unexpectedly shadow a stdlib one, but at least the stdlib itself doesn't have any conflicts. I should not ever have to do this dance: #include #undef SOME_SYMBOL #include But sadly, I do. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On 02/27/2015 04:21 PM, Travis Griggs wrote: On Feb 25, 2015, at 12:45 PM, Mark Lawrence wrote: http://www.slideshare.net/pydanny/python-worst-practices Any that should be added to this list? Any that be removed as not that bad? I read ‘em. I thought they were pretty good, some more than others. And I learned some things. I especially liked the first one, since I’ve struggled with that one a bunch. In the context of “hey, accept Python for what it is”, I agree greatly with it. Memorize the builtins, and stay the heck away from them. I’ve been burned many times because I stored some bytes in a binding call, er, uh, ‘bytes’. And having mentored some people learning Python in the early stages, any explanation other than “Thou Shalt Never Use These Holy Words” just confuses people. That said… If I were giving a talk at SPLASH (or some other suitable polyglot conference), I might do one called “Language Design Worst Practices”. One of my first slides might be titled: Abuse Common Tokens in Confusing Ways * Make your language have a lot of keywords. Enough to make memorizing them ALL unlikely, requiring constant visits to your documentation * Make sure said keywords are many of the obvious words programmers would use in their applications (map, object, bytes, dir, etc) * Design your syntax so that you can’t disambiguate them contextually between bind and reference * Be sure to use it in a late bound language where no warnings will be provided about the mistake you’re making at authorship time, deferring the educational experience to sundry run times In my examples column of this bad practice, I’d put Python of course. :) I do like Python, and I accept it for what it is, so no one needs to jump forward as a Holy Python See to convert me to the truth. I also know that with most other languages, that first slide wouldn’t need to be one of the prominent “worst practices” slide. Right. In C and C++, instead of being the first slide, it'd be the first 3 or 4. Between header file conflicts (especially good because the stdlib itself has many multiply-defined symbols, duplicate header files, and contradictory include path patterns), implib conflicts, and DLL load conflicts, not to mention the notorious win32 directory, which is magically renamed out from under you in 64bit code. (That one is a Windows thing; I wouldn't expect that Linux got caught in that trap) I don't know how much of this has changed in recent years, but I suspect it's gotten much worse, not better. In Python, the error is noticed much later, but it can actually be debugged without tearing out one's hair. Things like print(mymodule.__file__) to figure out where the mymodule is defined. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On Sat, Feb 28, 2015 at 8:21 AM, Travis Griggs wrote: > > I do like Python, and I accept it for what it is, so no one needs to jump > forward as a Holy Python See to convert me to the truth. I also know that > with most other languages, that first slide wouldn’t need to be one of the > prominent “worst practices” slide. > This is the thing about design. We are happy to rant about the problems in the things we love most... and it doesn't mean we don't love them. A passing comment in the famous "PHP: a fractal of bad design" article says: "Side observation: I lve Python. I will also happily talk your ear off complaining about it, if you really want me to."; for myself, I often say that Python is the second-best programming language in the world, but will still rant about its problems. And if I ever start building my own programming language, I would be looking at all the ones I know, and making objective evaluations of what's good and bad about them. (And then probably abandoning the project early on, because there are established languages that are sufficiently close to what I want that it's just not worth making a new one.) Know what you hate about the things you love, and know what you love about the things you hate. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
> On Feb 25, 2015, at 12:45 PM, Mark Lawrence wrote: > > http://www.slideshare.net/pydanny/python-worst-practices > > Any that should be added to this list? Any that be removed as not that bad? I read ‘em. I thought they were pretty good, some more than others. And I learned some things. I especially liked the first one, since I’ve struggled with that one a bunch. In the context of “hey, accept Python for what it is”, I agree greatly with it. Memorize the builtins, and stay the heck away from them. I’ve been burned many times because I stored some bytes in a binding call, er, uh, ‘bytes’. And having mentored some people learning Python in the early stages, any explanation other than “Thou Shalt Never Use These Holy Words” just confuses people. That said… If I were giving a talk at SPLASH (or some other suitable polyglot conference), I might do one called “Language Design Worst Practices”. One of my first slides might be titled: Abuse Common Tokens in Confusing Ways * Make your language have a lot of keywords. Enough to make memorizing them ALL unlikely, requiring constant visits to your documentation * Make sure said keywords are many of the obvious words programmers would use in their applications (map, object, bytes, dir, etc) * Design your syntax so that you can’t disambiguate them contextually between bind and reference * Be sure to use it in a late bound language where no warnings will be provided about the mistake you’re making at authorship time, deferring the educational experience to sundry run times In my examples column of this bad practice, I’d put Python of course. :) I do like Python, and I accept it for what it is, so no one needs to jump forward as a Holy Python See to convert me to the truth. I also know that with most other languages, that first slide wouldn’t need to be one of the prominent “worst practices” slide. -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On Sat, Feb 28, 2015 at 7:52 AM, Dave Angel wrote: > If that's the case on the architectures you're talking about, then the > problem of slow loading is not triggered by the memory usage, but by lots of > initialization code. THAT's what should be deferred for seldom-used > portions of code. s/should/can/ It's still not a clear case of "should", as it's all a big pile of trade-offs. A few weeks ago I made a very deliberate change to a process to force some code to get loaded and initialized earlier, to prevent an unexpected (and thus surprising) slowdown on first use. (It was, in fact, a Python 'import' statement, so all I had to do was add a dummy import in the main module - with, of course, a comment making it clear that this was necessary, even though the name wasn't used.) But yes, seldom-used code can definitely have its initialization deferred if you need to speed up startup. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On 02/27/2015 11:00 AM, alister wrote:
On Sat, 28 Feb 2015 01:22:15 +1100, Chris Angelico wrote:
If you're trying to use the pagefile/swapfile as if it's more memory ("I
have 256MB of memory, but 10GB of swap space, so that's 10GB of
memory!"), then yes, these performance considerations are huge. But
suppose you need to run a program that's larger than your available RAM.
On MS-DOS, sometimes you'd need to work with program overlays (a concept
borrowed from older systems, but ones that I never worked on, so I'm
going back no further than DOS here). You get a *massive* complexity hit
the instant you start using them, whether your program would have been
able to fit into memory on some systems or not. Just making it possible
to have only part of your code in memory places demands on your code
that you, the programmer, have to think about. With virtual memory,
though, you just write your code as if it's all in memory, and some of
it may, at some times, be on disk. Less code to debug = less time spent
debugging. The performance question is largely immaterial (you'll be
using the disk either way), but the savings on complexity are
tremendous. And then when you do find yourself running on a system with
enough RAM? No code changes needed, and full performance. That's where
virtual memory shines.
ChrisA
I think there is a case for bringing back the overlay file, or at least
loading larger programs in sections
only loading the routines as they are required could speed up the start
time of many large applications.
examples libre office, I rarely need the mail merge function, the word
count and may other features that could be added into the running
application on demand rather than all at once.
obviously with large memory & virtual mem there is no need to un-install
them once loaded.
I can't say how Linux handles it (I'd like to know, but haven't needed
to yet), but in Windows (NT, XP, etc), a DLL is not "loaded", but rather
mapped. And it's not copied into the swapfile, it's mapped directly
from the DLL. The mapping mode is "copy-on-write" which means that
read=only portions are swapped directly from the DLL, on first usage,
while read-write portions (eg. static/global variables, relocation
modifications) are copied on first use to the swap file. I presume
EXE's are done the same way, but never had a need to know.
If that's the case on the architectures you're talking about, then the
problem of slow loading is not triggered by the memory usage, but by
lots of initialization code. THAT's what should be deferred for
seldom-used portions of code.
The main point of a working-set-tuner is to group sections of code
together that are likely to be used together. To take an extreme case,
all the fatal exception handlers should be positioned adjacent to each
other in linear memory, as it's unlikely that any of them will be
needed, and the code takes up no time or space in physical memory.
Also (in Windows), a DLL can be pre-relocated, so that it has a
preferred address to be loaded into memory. If that memory is available
when it gets loaded (actually mapped), then no relocation needs to
happen, which saves time and swap space.
In the X86 architecture, most code is self-relocating, everything is
relative. But references to other DLL's and jump tables were absolute,
so they needed to be relocated at load time, when final locations were
nailed down.
Perhaps the authors of bloated applications have forgotten how to do
these, as the defaults in the linker puts all DLL's in the same
location, meaning all but the first will need relocating. But system
DLL's are (were) each given unique addresses.
On one large project, I added the build step of assigning these base
addresses. Each DLL had to start on a 64k boundary, and I reserved some
fractional extra space between them in case one would grow. Then every
few months, we double-checked that they didn't overlap, and if necessary
adjusted the start addresses. We didn't just automatically assign
closest addresses, because frequently some of the DLL's would be updated
independently of the others.
--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pyston 0.3 self-hosting
On 02/27/2015 12:41 PM, Travis Griggs wrote: > >> On Feb 24, 2015, at 9:47 PM, Steven D'Aprano >> wrote: >> >> Pyston 0.3, the latest version of a new high-performance Python >> implementation, has reached self-hosting sufficiency: >> >> >> http://blog.pyston.org/2015/02/24/pyston-0-3-self-hosting-sufficiency/ From their github site [1]: Currently, Pyston targets Python 2.7, only runs on x86_64 platforms, and only has been tested on Ubuntu. Support for more platforms -- along with Python 3 compatibility -- is desired but currently not on the roadmap. -- ~Ethan~ [1] https://github.com/dropbox/pyston signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyston 0.3 self-hosting
> On Feb 24, 2015, at 9:47 PM, Steven D'Aprano > wrote: > > Pyston 0.3, the latest version of a new high-performance Python > implementation, has reached self-hosting sufficiency: > > > http://blog.pyston.org/2015/02/24/pyston-0-3-self-hosting-sufficiency/ > Does it do python3.4 yet? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On 27 February 2015 20:06:25 GMT+00:00, I wrote: >I mentioned the true and false. OK, so it's a meme, but it's based on a >false (pun intended) understanding of exit status codes. That success >evaluates to true and failure evaluates to false does not mean the >values of truth and falseness are inverted. No programming language >other than that provided by system shells I have used evaluates 0 to >true. I hope the following examples from bash illustrate this: $ (( 0 )) && echo success $ (( 1 )) && echo success success $ (( 0 )) ; echo $? 1 $ (( 1 )) ; echo $? 0 Simon -- https://mail.python.org/mailman/listinfo/python-list
Re: function inclusion problem
On Wednesday, February 11, 2015 at 1:38:12 AM UTC+2, vlya...@gmail.com wrote:
> I defined function Fatalln in "mydef.py" and it works fine if i call it from
> "mydef.py", but when i try to call it from "test.py" in the same folder:
> import mydef
> ...
> Fatalln "my test"
> i have NameError: name 'Fatalln' is not defined
> I also tried include('mydef.py') with the same result...
> What is the right syntax?
> Thanks
...try to set your python utf-8 encode .
and read the FAQ or python manual
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On 26 February 2015 21:23:34 GMT+00:00, Ben Finney wrote: >Simon Ward writes: >> 0 = success and non-zero = failure is the meme established, rather >> than 0 = true, non-zero = false. > >That is not the case: the commands ‘true’ (returns value 0) and ‘false’ >(returns value 1) are long established in Unix. So that *is* part of >the >meme I'm describing. I mentioned the true and false. OK, so it's a meme, but it's based on a false (pun intended) understanding of exit status codes. That success evaluates to true and failure evaluates to false does not mean the values of truth and falseness are inverted. No programming language other than that provided by system shells I have used evaluates 0 to true. The shell case is unique because it typically runs processes and has to deal with exit status codes. It feels quite natural to me that, if COMMAND completes successfully (exit code 0) then writing "if COMMAND ; then : true bit ; else : false bit ; fi" should execute the true bit, but that doesn't mean all boolean logic is inverted. true is define to return 0, or no error. false is defined to return 1, or error. They are just commands though, and the exit codes are still exit codes, not boolean values. >> None of the above is a good reason to use error *or* success return >> values in Python--use exceptions!--but may be encountered when >running >> other processes. > >Right. But likewise, don't deny that “true == 0” and “false == >non-zero” >has a wide acceptance in the programming community too. I can't deny that, but I can state what things really mean and point out why everybody is doing it wrong. Simon -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On 2015-02-27 16:45, alister wrote: On Sat, 28 Feb 2015 03:12:16 +1100, Chris Angelico wrote: On Sat, Feb 28, 2015 at 3:00 AM, alister wrote: I think there is a case for bringing back the overlay file, or at least loading larger programs in sections only loading the routines as they are required could speed up the start time of many large applications. examples libre office, I rarely need the mail merge function, the word count and may other features that could be added into the running application on demand rather than all at once. Downside of that is twofold: firstly the complexity that I already mentioned, and secondly you pay the startup cost on first usage. So you might get into the program a bit faster, but as soon as you go to any feature you didn't already hit this session, the program pauses for a bit and loads it. Sometimes startup cost is the best time to do this sort of thing. If the modules are small enough this may not be noticeable but yes I do accept there may be delays on first usage. I suppose you could load the basic parts first so that the user can start working, and then load the additional features in the background. As to the complexity it has been my observation that as the memory footprint available to programmers has increase they have become less & less skilled at writing code. of course my time as a professional programmer was over 20 years ago on 8 bit micro controllers with 8k of ROM (eventually, original I only had 2k to play with) & 128 Bytes (yes bytes!) of RAM so I am very out of date. I now play with python because it is so much less demanding of me which probably makes me just a guilty :-) Of course, there is an easy way to implement exactly what you're asking for: use separate programs for everything, instead of expecting a megantic office suite[1] to do everything for you. Just get yourself a nice simple text editor, then invoke other programs - maybe from a terminal, or maybe from within the editor - to do the rest of the work. A simple disk cache will mean that previously-used programs start up quickly. Libre office was sighted as just one example Video editing suites are another that could be used as an example (perhaps more so, does the rendering engine need to be loaded until you start generating the output? a small delay here would be insignificant) ChrisA [1] It's slightly less bloated than the gigantic office suite sold by a top-end software company. -- https://mail.python.org/mailman/listinfo/python-list
Re: Parallelization of Python on GPU?
Am 26.02.15 um 06:53 schrieb John Ladasky: > Thanks for the various links, Ethan. I have encountered PyCUDA before, but > not the other options. > > So far, I'm not seeing code examples which appear to do what I would like, > which is simply to farm out one Python process to one GPU core. The examples > all appear to parallelize array operations. I know, that's the easier way to > break up a task. > > I may have to bite the bullet and learn how to use this: > > http://mklab.iti.gr/project/GPU-LIBSVM > If you can get this to run on your machine, it will surely outperform any efforts what you can do with a python-CUDA bridge on your own. GPU programming is hard, and efficient GPU programming is really hard. To get an impression, this talk shows how some changes to an OpenCL program can improve the speed by 60x compared to a naive implementation: http://web.archive.org/web/20101217181349/http://developer.amd.com/zones/OpenCLZone/Events/assets/Optimizations-ImageConvolution1.pdf Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: Gaussian process regression
On 27.02.2015 18:55, Peter Pearson wrote: On Thu, 26 Feb 2015 09:59:45 -0800 (PST),jaykim.hui...@gmail.com wrote: > >I am trying to use Gaussian process regression for Near Infrared >spectra. I have reference data(spectra), concentrations of reference >data and sample data, and I am trying to predict concentrations of >sample data. Here is my code. > >from sklearn.gaussian_process import GaussianProcess >gp = GaussianProcess() >gp.fit(reference, concentration) >concentration_pred = gp.predict(sample) [snip] I'm sorry you're not getting help from this normally very helpful group. I'd guess that's because nobody here uses sklearn. Where did you get sklearn? Is it possible that there's an sklearn forum somewhere? http://blog.gmane.org/gmane.comp.python.scikit-learn Cheers, Fabien -- https://mail.python.org/mailman/listinfo/python-list
Re: Gaussian process regression
On Thu, 26 Feb 2015 09:59:45 -0800 (PST), jaykim.hui...@gmail.com wrote: > > I am trying to use Gaussian process regression for Near Infrared > spectra. I have reference data(spectra), concentrations of reference > data and sample data, and I am trying to predict concentrations of > sample data. Here is my code. > > from sklearn.gaussian_process import GaussianProcess > gp = GaussianProcess() > gp.fit(reference, concentration) > concentration_pred = gp.predict(sample) [snip] I'm sorry you're not getting help from this normally very helpful group. I'd guess that's because nobody here uses sklearn. Where did you get sklearn? Is it possible that there's an sklearn forum somewhere? I've seen many of this group's regular participants go to great lengths to help people with specialized problems, but for one of those people to help with your problem, he or she would have to find and install sklearn and learn enough about it to generate data sets on which to exercise the code you've provided. That's a lot to ask. Can you lower the activation barrier? -- To email me, substitute nowhere->runbox, invalid->com. -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On 2015-02-27, Grant Edwards wrote: > On 2015-02-27, Steven D'Aprano wrote: > Dave Angel wrote: >>> On 02/27/2015 12:58 AM, Steven D'Aprano wrote: Dave Angel wrote: > (Although I believe Seymour Cray was quoted as saying that virtual > memory is a crock, because "you can't fake what you ain't got.") If I recall correctly, disk access is about 1 times slower than RAM, so virtual memory is *at least* that much slower than real memory. >>> It's so much more complicated than that, that I hardly know where to >>> start. >> >> [snip technical details] >> >> As interesting as they were, none of those details will make swap faster, >> hence my comment that virtual memory is *at least* 1 times slower than >> RAM. > > Nonsense. On all of my machines, virtual memory _is_ RAM almost all > of the time. I don't do the type of things that force the usage of > swap. And on some of the embedded systems I work on, _all_ virtual memory is RAM 100.000% of the time. -- Grant Edwards grant.b.edwardsYow! Don't SANFORIZE me!! at gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On 2015-02-27, Steven D'Aprano wrote: > Dave Angel wrote: > >> On 02/27/2015 12:58 AM, Steven D'Aprano wrote: >>> Dave Angel wrote: >>> (Although I believe Seymour Cray was quoted as saying that virtual memory is a crock, because "you can't fake what you ain't got.") >>> >>> If I recall correctly, disk access is about 1 times slower than RAM, >>> so virtual memory is *at least* that much slower than real memory. >>> >> >> It's so much more complicated than that, that I hardly know where to >> start. > > [snip technical details] > > As interesting as they were, none of those details will make swap faster, > hence my comment that virtual memory is *at least* 1 times slower than > RAM. Nonsense. On all of my machines, virtual memory _is_ RAM almost all of the time. I don't do the type of things that force the usage of swap. -- Grant Edwards grant.b.edwardsYow! ... I want FORTY-TWO at TRYNEL FLOATATION SYSTEMS gmail.cominstalled within SIX AND A HALF HOURS!!! -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On Sat, Feb 28, 2015 at 3:45 AM, alister wrote: > On Sat, 28 Feb 2015 03:12:16 +1100, Chris Angelico wrote: > >> On Sat, Feb 28, 2015 at 3:00 AM, alister >> wrote: >>> I think there is a case for bringing back the overlay file, or at least >>> loading larger programs in sections only loading the routines as they >>> are required could speed up the start time of many large applications. >>> examples libre office, I rarely need the mail merge function, the word >>> count and may other features that could be added into the running >>> application on demand rather than all at once. >> >> Downside of that is twofold: firstly the complexity that I already >> mentioned, and secondly you pay the startup cost on first usage. So you >> might get into the program a bit faster, but as soon as you go to any >> feature you didn't already hit this session, the program pauses for a >> bit and loads it. Sometimes startup cost is the best time to do this >> sort of thing. >> > If the modules are small enough this may not be noticeable but yes I do > accept there may be delays on first usage. > > As to the complexity it has been my observation that as the memory > footprint available to programmers has increase they have become less & > less skilled at writing code. Perhaps, but on the other hand, the skill of squeezing code into less memory is being replaced by other skills. We can write code that takes the simple/dumb approach, let it use an entire megabyte of memory, and not care about the cost... and we can write that in an hour, instead of spending a week fiddling with it. Reducing the development cycle time means we can add all sorts of cool features to a program, all while the original end user is still excited about it. (Of course, a comparison between today's World Wide Web and that of the 1990s suggests that these cool features aren't necessarily beneficial, but still, we have the option of foregoing austerity.) > Video editing suites are another that could be used as an example > (perhaps more so, does the rendering engine need to be loaded until you > start generating the output? a small delay here would be insignificant) Hmm, I'm not sure that's actually a big deal, because your *data* will dwarf the code. I can fire up sox and avconv, both fairly large programs, and their code will all sit comfortably in memory; but then they get to work on my data, and suddenly my hard disk is chewing through 91GB of content. Breaking up avconv into a dozen pieces wouldn't make a dent in 91GB. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On Sat, 28 Feb 2015 03:12:16 +1100, Chris Angelico wrote: > On Sat, Feb 28, 2015 at 3:00 AM, alister > wrote: >> I think there is a case for bringing back the overlay file, or at least >> loading larger programs in sections only loading the routines as they >> are required could speed up the start time of many large applications. >> examples libre office, I rarely need the mail merge function, the word >> count and may other features that could be added into the running >> application on demand rather than all at once. > > Downside of that is twofold: firstly the complexity that I already > mentioned, and secondly you pay the startup cost on first usage. So you > might get into the program a bit faster, but as soon as you go to any > feature you didn't already hit this session, the program pauses for a > bit and loads it. Sometimes startup cost is the best time to do this > sort of thing. > If the modules are small enough this may not be noticeable but yes I do accept there may be delays on first usage. As to the complexity it has been my observation that as the memory footprint available to programmers has increase they have become less & less skilled at writing code. of course my time as a professional programmer was over 20 years ago on 8 bit micro controllers with 8k of ROM (eventually, original I only had 2k to play with) & 128 Bytes (yes bytes!) of RAM so I am very out of date. I now play with python because it is so much less demanding of me which probably makes me just a guilty :-) > Of course, there is an easy way to implement exactly what you're asking > for: use separate programs for everything, instead of expecting a > megantic office suite[1] to do everything for you. Just get yourself a > nice simple text editor, then invoke other programs - maybe from a > terminal, or maybe from within the editor - to do the rest of the work. > A simple disk cache will mean that previously-used programs start up > quickly. Libre office was sighted as just one example Video editing suites are another that could be used as an example (perhaps more so, does the rendering engine need to be loaded until you start generating the output? a small delay here would be insignificant) > > ChrisA > > [1] It's slightly less bloated than the gigantic office suite sold by a > top-end software company. -- You don't sew with a fork, so I see no reason to eat with knitting needles. -- Miss Piggy, on eating Chinese Food -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On Sat, Feb 28, 2015 at 3:00 AM, alister wrote: > I think there is a case for bringing back the overlay file, or at least > loading larger programs in sections > only loading the routines as they are required could speed up the start > time of many large applications. > examples libre office, I rarely need the mail merge function, the word > count and may other features that could be added into the running > application on demand rather than all at once. Downside of that is twofold: firstly the complexity that I already mentioned, and secondly you pay the startup cost on first usage. So you might get into the program a bit faster, but as soon as you go to any feature you didn't already hit this session, the program pauses for a bit and loads it. Sometimes startup cost is the best time to do this sort of thing. Of course, there is an easy way to implement exactly what you're asking for: use separate programs for everything, instead of expecting a megantic office suite[1] to do everything for you. Just get yourself a nice simple text editor, then invoke other programs - maybe from a terminal, or maybe from within the editor - to do the rest of the work. A simple disk cache will mean that previously-used programs start up quickly. ChrisA [1] It's slightly less bloated than the gigantic office suite sold by a top-end software company. -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On Sat, 28 Feb 2015 01:22:15 +1100, Chris Angelico wrote:
>
> If you're trying to use the pagefile/swapfile as if it's more memory ("I
> have 256MB of memory, but 10GB of swap space, so that's 10GB of
> memory!"), then yes, these performance considerations are huge. But
> suppose you need to run a program that's larger than your available RAM.
> On MS-DOS, sometimes you'd need to work with program overlays (a concept
> borrowed from older systems, but ones that I never worked on, so I'm
> going back no further than DOS here). You get a *massive* complexity hit
> the instant you start using them, whether your program would have been
> able to fit into memory on some systems or not. Just making it possible
> to have only part of your code in memory places demands on your code
> that you, the programmer, have to think about. With virtual memory,
> though, you just write your code as if it's all in memory, and some of
> it may, at some times, be on disk. Less code to debug = less time spent
> debugging. The performance question is largely immaterial (you'll be
> using the disk either way), but the savings on complexity are
> tremendous. And then when you do find yourself running on a system with
> enough RAM? No code changes needed, and full performance. That's where
> virtual memory shines.
> ChrisA
I think there is a case for bringing back the overlay file, or at least
loading larger programs in sections
only loading the routines as they are required could speed up the start
time of many large applications.
examples libre office, I rarely need the mail merge function, the word
count and may other features that could be added into the running
application on demand rather than all at once.
obviously with large memory & virtual mem there is no need to un-install
them once loaded.
--
Ralph's Observation:
It is a mistake to let any mechanical object realise that you
are in a hurry.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On 02/27/2015 09:22 AM, Chris Angelico wrote:
On Sat, Feb 28, 2015 at 1:02 AM, Dave Angel wrote:
The term "virtual memory" is used for many aspects of the modern memory
architecture. But I presume you're using it in the sense of "running in a
swapfile" as opposed to running in physical RAM.
Given that this started with a quote about "you can't fake what you
ain't got", I would say that, yes, this refers to using hard disk to
provide more RAM.
If you're trying to use the pagefile/swapfile as if it's more memory
("I have 256MB of memory, but 10GB of swap space, so that's 10GB of
memory!"), then yes, these performance considerations are huge. But
suppose you need to run a program that's larger than your available
RAM. On MS-DOS, sometimes you'd need to work with program overlays (a
concept borrowed from older systems, but ones that I never worked on,
so I'm going back no further than DOS here). You get a *massive*
complexity hit the instant you start using them, whether your program
would have been able to fit into memory on some systems or not. Just
making it possible to have only part of your code in memory places
demands on your code that you, the programmer, have to think about.
With virtual memory, though, you just write your code as if it's all
in memory, and some of it may, at some times, be on disk. Less code to
debug = less time spent debugging. The performance question is largely
immaterial (you'll be using the disk either way), but the savings on
complexity are tremendous. And then when you do find yourself running
on a system with enough RAM? No code changes needed, and full
performance. That's where virtual memory shines.
It's funny how the world changes, though. Back in the 90s, virtual
memory was the key. No home computer ever had enough RAM. Today? A
home-grade PC could easily have 16GB... and chances are you don't need
all of that. So we go for the opposite optimization: disk caching.
Apart from when I rebuild my "Audio-Only Frozen" project [1] and the
caches get completely blasted through, heaps and heaps of my work can
be done inside the disk cache. Hey, Sikorsky, got any files anywhere
on the hard disk matching *Pastel*.iso case insensitively? *chug chug
chug* Nope. Okay. Sikorsky, got any files matching *Pas5*.iso case
insensitively? *zip* Yeah, here it is. I didn't tell the first search
to hold all that file system data in memory; the hard drive controller
managed it all for me, and I got the performance benefit. Same as the
above: the main benefit is that this sort of thing requires zero
application code complexity. It's all done in a perfectly generic way
at a lower level.
In 1973, I did manual swapping to an external 8k ramdisk. It was a box
that sat on the floor and contained 8k of core memory (not
semiconductor). The memory was non-volatile, so it contained the
working copy of my code. Then I built a small swapper that would bring
in the set of routines currently needed. My onboard RAM (semiconductor)
was 1.5k, which had to hold the swapper, the code, and the data. I was
writing a GPS system for shipboard use, and the final version of the
code had to fit entirely in EPROM, 2k of it. But debugging EPROM code
is a pain, since every small change took half an hour to make new chips.
Later, I built my first PC with 512k of RAM, and usually used much of it
as a ramdisk, since programs didn't use nearly that amount.
--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On Sat, Feb 28, 2015 at 1:02 AM, Dave Angel wrote:
> The term "virtual memory" is used for many aspects of the modern memory
> architecture. But I presume you're using it in the sense of "running in a
> swapfile" as opposed to running in physical RAM.
Given that this started with a quote about "you can't fake what you
ain't got", I would say that, yes, this refers to using hard disk to
provide more RAM.
If you're trying to use the pagefile/swapfile as if it's more memory
("I have 256MB of memory, but 10GB of swap space, so that's 10GB of
memory!"), then yes, these performance considerations are huge. But
suppose you need to run a program that's larger than your available
RAM. On MS-DOS, sometimes you'd need to work with program overlays (a
concept borrowed from older systems, but ones that I never worked on,
so I'm going back no further than DOS here). You get a *massive*
complexity hit the instant you start using them, whether your program
would have been able to fit into memory on some systems or not. Just
making it possible to have only part of your code in memory places
demands on your code that you, the programmer, have to think about.
With virtual memory, though, you just write your code as if it's all
in memory, and some of it may, at some times, be on disk. Less code to
debug = less time spent debugging. The performance question is largely
immaterial (you'll be using the disk either way), but the savings on
complexity are tremendous. And then when you do find yourself running
on a system with enough RAM? No code changes needed, and full
performance. That's where virtual memory shines.
It's funny how the world changes, though. Back in the 90s, virtual
memory was the key. No home computer ever had enough RAM. Today? A
home-grade PC could easily have 16GB... and chances are you don't need
all of that. So we go for the opposite optimization: disk caching.
Apart from when I rebuild my "Audio-Only Frozen" project [1] and the
caches get completely blasted through, heaps and heaps of my work can
be done inside the disk cache. Hey, Sikorsky, got any files anywhere
on the hard disk matching *Pastel*.iso case insensitively? *chug chug
chug* Nope. Okay. Sikorsky, got any files matching *Pas5*.iso case
insensitively? *zip* Yeah, here it is. I didn't tell the first search
to hold all that file system data in memory; the hard drive controller
managed it all for me, and I got the performance benefit. Same as the
above: the main benefit is that this sort of thing requires zero
application code complexity. It's all done in a perfectly generic way
at a lower level.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
On 02/27/2015 06:54 AM, Steven D'Aprano wrote: Dave Angel wrote: On 02/27/2015 12:58 AM, Steven D'Aprano wrote: Dave Angel wrote: (Although I believe Seymour Cray was quoted as saying that virtual memory is a crock, because "you can't fake what you ain't got.") If I recall correctly, disk access is about 1 times slower than RAM, so virtual memory is *at least* that much slower than real memory. It's so much more complicated than that, that I hardly know where to start. [snip technical details] As interesting as they were, none of those details will make swap faster, hence my comment that virtual memory is *at least* 1 times slower than RAM. The term "virtual memory" is used for many aspects of the modern memory architecture. But I presume you're using it in the sense of "running in a swapfile" as opposed to running in physical RAM. Yes, a page fault takes on the order of 10,000 times as long as an access to a location in L1 cache. I suspect it's a lot smaller though if the swapfile is on an SSD drive. The first byte is that slow. But once the fault is resolved, the nearby bytes are in physical memory, and some of them are in L3, L2, and L1. So you're not running in the swapfile any more. And even when you run off the end of the page, fetching the sequentially adjacent page from a hard disk is much faster. And if the disk has well designed buffering, faster yet. The OS tries pretty hard to keep the swapfile unfragmented. The trick is to minimize the number of page faults, especially to random locations. If you're getting lots of them, it's called thrashing. There are tools to help with that. To minimize page faults on code, linking with a good working-set-tuner can help, though I don't hear of people bothering these days. To minimize page faults on data, choosing one's algorithm carefully can help. For example, in scanning through a typical matrix, row order might be adjacent locations, while column order might be scattered. Not really much different than reading a text file. If you can arrange to process it a line at a time, rather than reading the whole file into memory, you generally minimize your round-trips to disk. And if you need to randomly access it, it's quite likely more efficient to memory map it, in which case it temporarily becomes part of the swapfile system. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about text encoding
Dave Angel wrote: > On 02/27/2015 12:58 AM, Steven D'Aprano wrote: >> Dave Angel wrote: >> >>> (Although I believe Seymour Cray was quoted as saying that virtual >>> memory is a crock, because "you can't fake what you ain't got.") >> >> If I recall correctly, disk access is about 1 times slower than RAM, >> so virtual memory is *at least* that much slower than real memory. >> > > It's so much more complicated than that, that I hardly know where to > start. [snip technical details] As interesting as they were, none of those details will make swap faster, hence my comment that virtual memory is *at least* 1 times slower than RAM. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
please help to run pylearn2
I try to run pylearn2 tutorial: Softmax regression using my notebook.
but i run
from pylearn2.config import yaml_parse
train = yaml_parse.load(train)
train.main_loop()
this code in my notebook i got Error.How to solve this
Please help
---
ImportError Traceback (most recent call last)
in ()
1 from pylearn2.config import yaml_parse
> 2 train = yaml_parse.load(train)
3 train.main_loop()
/home/amila/Documents/Algorithm/git/pylearn2/pylearn2/config/yaml_parse.pyc in
load(stream, environ, instantiate, **kwargs)
209 string = stream.read()
210
--> 211 proxy_graph = yaml.load(string, **kwargs)
212 if instantiate:
213 return _instantiate(proxy_graph)
/usr/lib/python2.7/dist-packages/yaml/__init__.pyc in load(stream, Loader)
69 loader = Loader(stream)
70 try:
---> 71 return loader.get_single_data()
72 finally:
73 loader.dispose()
/usr/lib/python2.7/dist-packages/yaml/constructor.pyc in get_single_data(self)
37 node = self.get_single_node()
38 if node is not None:
---> 39 return self.construct_document(node)
40 return None
41
/usr/lib/python2.7/dist-packages/yaml/constructor.pyc in
construct_document(self, node)
41
42 def construct_document(self, node):
---> 43 data = self.construct_object(node)
44 while self.state_generators:
45 state_generators = self.state_generators
/usr/lib/python2.7/dist-packages/yaml/constructor.pyc in construct_object(self,
node, deep)
88 data = constructor(self, node)
89 else:
---> 90 data = constructor(self, tag_suffix, node)
91 if isinstance(data, types.GeneratorType):
92 generator = data
/home/amila/Documents/Algorithm/git/pylearn2/pylearn2/config/yaml_parse.pyc in
multi_constructor_obj(loader, tag_suffix, node)
356 yaml_src = yaml.serialize(node)
357 construct_mapping(node)
--> 358 mapping = loader.construct_mapping(node)
359
360 assert hasattr(mapping, 'keys')
/usr/lib/python2.7/dist-packages/yaml/constructor.pyc in
construct_mapping(self, node, deep)
206 if isinstance(node, MappingNode):
207 self.flatten_mapping(node)
--> 208 return BaseConstructor.construct_mapping(self, node, deep=deep)
209
210 def construct_yaml_null(self, node):
/usr/lib/python2.7/dist-packages/yaml/constructor.pyc in
construct_mapping(self, node, deep)
131 raise ConstructorError("while constructing a mapping",
node.start_mark,
132 "found unacceptable key (%s)" % exc,
key_node.start_mark)
--> 133 value = self.construct_object(value_node, deep=deep)
134 mapping[key] = value
135 return mapping
/usr/lib/python2.7/dist-packages/yaml/constructor.pyc in construct_object(self,
node, deep)
88 data = constructor(self, node)
89 else:
---> 90 data = constructor(self, tag_suffix, node)
91 if isinstance(data, types.GeneratorType):
92 generator = data
/home/amila/Documents/Algorithm/git/pylearn2/pylearn2/config/yaml_parse.pyc in
multi_constructor_obj(loader, tag_suffix, node)
370 callable = eval(tag_suffix)
371 else:
--> 372 callable = try_to_import(tag_suffix)
373 rval = Proxy(callable=callable, yaml_src=yaml_src, positionals=(),
374 keywords=mapping)
/home/amila/Documents/Algorithm/git/pylearn2/pylearn2/config/yaml_parse.pyc in
try_to_import(tag_suffix)
297 base_msg += ' but could import %s' % modulename
298 reraise_as(ImportError(base_msg + '. Original
exception: '
--> 299+ str(e)))
300 j += 1
301 try:
/home/amila/Documents/Algorithm/git/pylearn2/pylearn2/utils/exc.pyc in
reraise_as(new_exc)
88 new_exc.__cause__ = orig_exc_value
89 new_exc.reraised = True
---> 90 six.reraise(type(new_exc), new_exc, orig_exc_traceback)
/home/amila/Documents/Algorithm/git/pylearn2/pylearn2/config/yaml_parse.pyc in
try_to_import(tag_suffix)
290 modulename = '.'.join(pcomponents[:j])
291 try:
--> 292 exec('import %s' % modulename)
293 except Exception:
294 base_msg = 'Could not import %s' % modulename
/home/amila/Documents/Algorithm/git/pylearn2/pylearn2/config/yaml_parse.pyc in
()
/home/amila/Documents/Algorithm/git/pylearn2/pylearn2/models/softmax_regression.py
in ()
13 __maintainer__ = "LISA Lab"
14
---> 15 from pylearn2.models import mlp
16
17 class SoftmaxRegressio
Re: strip bug?
Hello how are you doing please -- https://mail.python.org/mailman/listinfo/python-list
Re: mx as dependency tox throws GCC error
On 25.02.2015 15:43, Albert-Jan Roskam wrote:
> Hi,
>
> If I pip install the mx package with "pip install egenix-mx-base", it works.
> If I put that same pip install command under 'install_command' in my tox.ini
> it also works (see below)
>
> However, if I specify the dependency under 'deps', I get an error. Any idea
> why? I read that 'install_command' is
> still experimental and subject to change, so I would prefer to specify the
> package under 'deps'.
> The error occurred when I called tox using 'tox -e py27', so it is not
> pypy-related.
>
>
> [tox]
> envlist = pypy,py27,py33,py34,docs
> skipsdist = true
>
> [testenv]
> deps =
>setuptools
> nose
> gocept.testing
> numpy
> cython
> # egenix-mx-base # --> mx tox gcc: error:
> mx/DateTime/mxDateTime/mxDateTime.c: No such file or directory
> install_command =
>pip install egenix-mx-base {packages}
Could you provide some more context or debug output ?
The .c file is definitely part of the egenix-mx-base package,
but it's possible that tox is using a different way of
installing the package than pip itself does when using
it under "deps".
If there's a way to work around the problem, we'll fix this
in our mxSetup.py.
Thanks,
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 27 2015)
>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
: Try our mxODBC.Connect Python Database Interface for free ! ::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python Worst Practices
On 27/02/2015 01:31, Dennis Lee Bieber wrote: On Thu, 26 Feb 2015 20:10:28 +, Simon Ward declaimed the following: 0 = success and non-zero = failure is the meme established, rather than 0 = true, non-zero = false. It's not just used by UNIX, and is not necessarily defined by the shell either (bash was mentioned elsewhere in the thread). There is probably a system that pre-dates UNIX that I uses/used this too, but I don't know. I miss VMS... Odd => true, even => false Odd -> success, +even -> warning, -even -> error (odd may have been success vs info> Me to, Very Much Safer :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
