Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.
On Fri, Feb 19, 2010 at 11:16 PM, Lie Ryan lie.1...@gmail.com wrote: Now, why don't we start a PEP to make python a fully-functional language then? Because people don't think the same way that programs are written in functional languages. -- Jonathan Gardner jgard...@jonathangardner.net -- http://mail.python.org/mailman/listinfo/python-list
Re: Capturing errors raised by other scripts ?
On 02/20/10 14:39, northof40 wrote: On Feb 20, 4:13 pm, MRAB pyt...@mrabarnett.plus.com wrote: northof40 wrote: I'm using the subroutine module to run run python script A.py from B.py (this is on windows fwiw). A.py is not my script and it may raise arbitary errors before exiting. How can I determine what's happened before A.py exited ? To simulate this I've got this script (which is meant to simulate A.py): class customError(Exception): def __init__(self, value): self.value = value def __str__(self): return repr(self.value) try: raise customError(2*2) except customError as e: print 'Custom exception occurred, value:', e.value I then run my A.py like this : fdOut, fOut = tempfile.mkstemp(suffix='.txt', prefix='AOut-') fdErr, fErr = tempfile.mkstemp(suffix='.txt', prefix='AErr-') try: ... pathtojob=python.exe A.py ... p = subprocess.Popen(pathtojob, stderr=fdErr, stdout=fdOut) ... except: ... print bad stuff happened ... When I do this I the exception handler is not fired and the text Custom exception occurred, value: 4 ends up in the stdout file. I'd really like it to end up in stderr because then I could say anything in stderr ? then ignore the output and flag an error. Not really; not all that is written to stderr signifies errors per se. Warning and Debugging info are often written to stderr as well. Thanks for your reply. I take your point about print - perhaps I hadn't really thought that through. Unfortunately I don't have control of the real script that's being run so if the programmer of that script has used print there's nothing I can do about it. Tracebacks is almost always written to stderr, not stdout; unless the programmer explicitly redirect the traceback into stdout. Perhaps I could modify the question. If A.py terminates due an unhandled exception is there anyway for B.py to know that's happened (without A.py's cooperation ?). A well-behaved program will exit with status code 0; if and only if it exited cleanly. I believe a python program that exited due to unhandled exception always return non-zero status code. However, if the program handled an error and called exit(0) explicitly or the execution of the program falls to the end of the module; there is no way to distinguish it from regular clean exit. -- http://mail.python.org/mailman/listinfo/python-list
Re: Can't Access ANY url from python (errno 61)
On Feb 20, 2:02 am, Lie Ryan lie.1...@gmail.com wrote: On 02/20/10 13:32, MattB wrote: I'm using the network in my own apartment. Not the campus's. Moreover, my mac's MAC address is different from the MAC address shown by my router, but as I said I'm also blocked when using my friend's wireless router at his apartment. So it must be my mac's MAC, and not the router's MAC, that's being blocked, right? But ALSO -- is it my ISP that's blocking the mac's MAC (and not the school), since I can't raise ANY url's from python when I'm on wireless? MAC or IP blocking can't be the reason, as the OP stated, he can use Firefox just fine. Can you access, say,http://www.google.comfrom urllib or mechanize? If you can't access *any website* using urllib/mechanize but you can with a browser and you're on a unix-based machine, you probably have the same problem as I used to have. Check whether you used the same hostname in /etc/conf.d/hostname and /etc/hosts (or wherever your distro saves its hostname configurations, I use Gentoo); after editing those files reboot (there are ways to avoid reboot, but rebooting guarantees the conf file is reread). Check the hostname by running this python script: import socket hn = socket.gethostname() print hn print socket.gethostbyname(hn) # should be 127.0.0.1 Lie, Wow. Strangely, that script returned 192.168.1.106. However, in Snow Leopard's airport settings, if I click on 'advanced' and then 'proxies', the default proxy for 'http' is 127.0.0.1: (and in these settings, the 'use proxy for http' is checked). I just tried checking the unix files you mentioned. In etc/hosts, the following info is displayed: ## # Host Database # # localhost is used to configure the loopback interface # when the system is booting. Do not change this entry. ## 127.0.0.1 localhost 255.255.255.255 broadcasthost ::1 localhost fe80::1%lo0 localhost Also found a file called ntp-restrict.conf, containing: # Access restrictions documented in ntp.conf(5) and # http://support.ntp.org/bin/view/Support/AccessRestrictions # Limit network machines to time queries only restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery # localhost is unrestricted restrict 127.0.0.1 restrict -6 ::1 includefile /private/etc/ntp.conf Not sure if these are what I'm looking for -- I'm new to unix so I may need a bit more hand-holding here. I appreciate your time and effort. Matt -- http://mail.python.org/mailman/listinfo/python-list
Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.
On Fri, Feb 19, 2010 at 11:17 PM, sjdevn...@yahoo.com sjdevn...@yahoo.com wrote: On Feb 20, 1:30 am, Lawrence D'Oliveiro l...@geek- central.gen.new_zealand wrote: If Python doesn’t distinguish between procedures and functions, why should it distinguish between statements and expressions? Because the latter are different in Python (and in Ruby I think your Ruby assertion needs fact-checking: irb(main):001:0 a = 7 # assignments have a value = 7 irb(main):002:0 puts(b = 42) # as further proof 42 = nil irb(main):003:0 b = 42 irb(main):004:0 c = [6,4,5] = [6, 4, 5] irb(main):005:0 if false irb(main):006:1 c.reverse! irb(main):007:1 else irb(main):008:1* c.sort! irb(main):009:1 end # even the if-else control structure has a value = [4, 5, 6] irb(main):010:0 begin # same with exception handling irb(main):011:1*raise a runtime error irb(main):012:1 rescue RuntimeError irb(main):013:1 sounds bad irb(main):014:1 end = sounds bad irb(main):015:0 def foo # and same with method bodies irb(main):016:1 99 irb(main):017:1 end = nil irb(main):018:0 foo = 99 Quoth Wikipedia regarding Ruby (programming language): For practical purposes there is no distinction between expressions and statements Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Can't Access ANY url from python (errno 61)
throw up your 'ifconfig' and mozilla-proxy output here. It seems you don't know whether you are using proxy. For mozilla proxy : open mozilla - cmd + , - Network - Settings - Paste everything that is there/ may be take a snapshot and upload a link. ~l0nwlf On Sat, Feb 20, 2010 at 2:06 PM, MattB mattbar...@gmail.com wrote: On Feb 20, 2:02 am, Lie Ryan lie.1...@gmail.com wrote: On 02/20/10 13:32, MattB wrote: I'm using the network in my own apartment. Not the campus's. Moreover, my mac's MAC address is different from the MAC address shown by my router, but as I said I'm also blocked when using my friend's wireless router at his apartment. So it must be my mac's MAC, and not the router's MAC, that's being blocked, right? But ALSO -- is it my ISP that's blocking the mac's MAC (and not the school), since I can't raise ANY url's from python when I'm on wireless? MAC or IP blocking can't be the reason, as the OP stated, he can use Firefox just fine. Can you access, say,http://www.google.comfrom urllib or mechanize? If you can't access *any website* using urllib/mechanize but you can with a browser and you're on a unix-based machine, you probably have the same problem as I used to have. Check whether you used the same hostname in /etc/conf.d/hostname and /etc/hosts (or wherever your distro saves its hostname configurations, I use Gentoo); after editing those files reboot (there are ways to avoid reboot, but rebooting guarantees the conf file is reread). Check the hostname by running this python script: import socket hn = socket.gethostname() print hn print socket.gethostbyname(hn) # should be 127.0.0.1 Lie, Wow. Strangely, that script returned 192.168.1.106. However, in Snow Leopard's airport settings, if I click on 'advanced' and then 'proxies', the default proxy for 'http' is 127.0.0.1: (and in these settings, the 'use proxy for http' is checked). I just tried checking the unix files you mentioned. In etc/hosts, the following info is displayed: ## # Host Database # # localhost is used to configure the loopback interface # when the system is booting. Do not change this entry. ## 127.0.0.1 localhost 255.255.255.255 broadcasthost ::1 localhost fe80::1%lo0 localhost Also found a file called ntp-restrict.conf, containing: # Access restrictions documented in ntp.conf(5) and # http://support.ntp.org/bin/view/Support/AccessRestrictions # Limit network machines to time queries only restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery # localhost is unrestricted restrict 127.0.0.1 restrict -6 ::1 includefile /private/etc/ntp.conf Not sure if these are what I'm looking for -- I'm new to unix so I may need a bit more hand-holding here. I appreciate your time and effort. Matt -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
os.pipe() + os.fork()
Hi, I have a problem using os.pipe() together with os.fork(). Usually when the writing end of the pipe is closed, the reading end gets EOF. So subsequent attempts to read data will return an empty string. But when you call os.fork() after you have created a pipe using os.pipe(), and read data from the pipe in the child process, when the partent process has already closed the writing end, the reading end does not get EOF. Instead of the os.read() call in the child process is blocking. I am using Linux, so I would like to know if this is a bug in Python or just weird behaviour under Linux and if it is possible work around it. Regards Sebastian Noack --- import os def test_pipe_sync(): pipe = os.pipe() os.write(pipe[1], 'Spam') os.close(pipe[1]) print repr(os.read(pipe[0], 4))# 'Spam' is printed. print repr(os.read(pipe[0], 4))# '' is printed, because of EOF is reached. def test_pipe_forked(): pipe = os.pipe() pid = os.fork() if pid: os.write(pipe[1], 'Spam') os.close(pipe[1]) os.waitpid(pid, 0) else: print repr(os.read(pipe[0], 4))# 'Spam' is printed. print repr(os.read(pipe[0], 4))# Nothing is printed and os.read() # is blocking, eventhough the other # end of the pipe was closed. if __name__ == '__main__': test_pipe_sync() print print '-' * 80 print test_pipe_forked() signature.asc Description: PGP signature -- http://mail.python.org/mailman/listinfo/python-list
Re: Avoid converting functions to methods in a class
On 20 Feb, 03:33, Steven D'Aprano st...@remove-this-
cybersource.com.au wrote:
I have a convention when writing unit tests to put the target of the test
into a class attribute, as follows:
class MyTest(unittest.TestCase):
target = mymodule.someclass
def test_spam(self):
Test that someclass has a spam attribute.
self.failUnless(hasattr(self.target, 'spam'))
It works well until I write a test for stand-alone functions:
class AnotherTest(unittest.TestCase):
target = mymodule.function
def test_foo(self):
self.assertEquals(self.target('a', 'b'), 'foo')
The problem is that target is turned into a method of my test class, not
a standalone function, and I get errors like:
TypeError: function() takes exactly 2 arguments (3 given)
The solution I currently use is to drop the target attribute in this
class, and just refer to mymodule.function in each individual test. I
don't like this solution because it violates Once And Only Once: if the
function changes name, I have to make many edits to the test suite rather
than just one.
Are there any better solutions?
--
Steven
Why not define target in the TestCase.setUp() method?
class AnotherTest(unittest.TestCase):
def setUp(self):
self.target = mymodule.function
def test_foo(self):
self.assertEquals(self.target('a', 'b'), 'foo')
--
Arnaud
class
--
Arnaud
--
http://mail.python.org/mailman/listinfo/python-list
Re: Scalable python dict {'key_is_a_string': [count, some_val]}
On 20 Feb, 06:36, krishna krishna.k.0...@gmail.com wrote: I have to manage a couple of dicts with huge dataset (larger than feasible with the memory on my system), it basically has a key which is a string (actually a tuple converted to a string) and a two item list as value, with one element in the list being a count related to the key. I have to at the end sort this dictionary by the count. The platform is linux. I am planning to implement it by setting a threshold beyond which I write the data into files (3 columns: 'key count some_val' ) and later merge those files (I plan to sort the individual files by the key column and walk through the files with one pointer per file and merge them; I would add up the counts when entries from two files match by key) and sorting using the 'sort' command. Thus the bottleneck is the 'sort' command. Any suggestions, comments? By the way, is there a linux command that does the merging part? Thanks, Krishna Have you looked here? http://docs.python.org/library/persistence.html -- Arnaud -- http://mail.python.org/mailman/listinfo/python-list
Precision issue in python
Hello everyone. I think it is related to the precision with double arithmetic so i posted here.I am trying with this problem (https:// www.spoj.pl/problems/CALCULAT) and the problem say that Note : for all test cases whose N=100, its K=15. I know precision of doubles in c is 16 digits. Could some one please help me with this precision issue.I used stirling (http://en.wikipedia.org/wiki/ Stirling's_approximation) to calculate the first k digits of N. Thank you. __author__=Administrator __date__ =$Feb 19, 2010 3:16:47 PM$ import math if __name__ == __main__: n=int(raw_input()) while(n0): n-=1 raw_input() l=raw_input(); m=l.split( ) N,K,L=int(m[0]),int(m[1]),int(m[2]) fwd,bkd=1,1 s_1,s_2=, if(N=200): for i in range(1,N+1): fwd=fwd*i; s=str(fwd) s_1=s[:K] else: d_1=(N*math.log(N)-N+0.5*math.log(2*math.pi*N)+(1/12/N)- (1/360/pow(N,3))+(1/1260/pow(N,5))-(1/1680/pow(N,7))+(1/1188/pow(N,9))- (691/2730/12/11/pow(N,11))+(7/6/14/13/pow(N,13)))*math.log10(math.e) d_2=d_1-int(d_1)+K-1 fwd=pow(10,d_2) #print fwd fwd=int(fwd) s_1=str(fwd) if(N=500): for i in range(1,N+1): bkd=bkd*i bkd=bkd%pow(10,L) if(bkd==0): s_2=0*L else: s_2=str(bkd) else: s_2=0*L print s_1+ +s_2 -- http://mail.python.org/mailman/listinfo/python-list
Shared web host good citizenship question
Not specifically Python related but thought I'd ask. If you have a low-bandwidth website running on a shared web hosting service as a WSGI server (or FastCGI, or any other interface with its own process), is it considered a responsible thing for the process to exit on its own after a period of non-activity, or is that something best left for the host to manage? I'd guess that modern web hosts would prefer that processes stay running, even if they go long periods without use, rather than having to periodically restart processes. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: os.pipe() + os.fork()
I have figured out that, you have to close the writing end in the child process, which is reading from the pipe. Otherwise the underlying pipe is not going to be closed when the parent process is closing its writing end. This has nothing to do with Python itself. I have tried plain C and there it is the same behaviour. Regards Sebastian Noack signature.asc Description: PGP signature -- http://mail.python.org/mailman/listinfo/python-list
Re: Precision issue in python
On Feb 20, 11:17 am, mukesh tiwari mukeshtiwari.ii...@gmail.com wrote: Hello everyone. I think it is related to the precision with double arithmetic so i posted here.I am trying with this problem (https://www.spoj.pl/problems/CALCULAT) and the problem say that Note : for all test cases whose N=100, its K=15. I know precision of doubles in c is 16 digits. Could some one please help me with this precision issue.I used stirling (http://en.wikipedia.org/wiki/ Stirling's_approximation) to calculate the first k digits of N. Thank you. If I understand you correctly, you're trying to compute the first k digits in the decimal expansion of N!, with bounds of k = 15 and 100 = N 10**8. Is that right? Python's floats simply don't give you enough precision for this: you'd need to find the fractional part of log10(N!) with = 15 digits of precision. But the integral part needs ~ log10(log10(N!)) digits, so you end up needing to compute log10(N!) with at least 15 + log10(log10(N!)) ~ 15 + log10(N) + log10(log10(N)) digits (plus a few extra digits for safety); so that's at least 25 digits needed for N close to 10**8. The decimal module would get you the results you need (are you allowed imports?). Or you could roll your own log implementation based on integer arithmetic. -- Mark -- http://mail.python.org/mailman/listinfo/python-list
Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.
On Feb 18, 4:15 pm, Steve Howell showel...@yahoo.com wrote:
...
def print_numbers()
[1, 2, 3, 4, 5, 6].map { |n|
[n * n, n * n * n]
}.reject { |square, cube|
square == 25 || cube == 64
}.map { |square, cube|
cube
}.each { |n|
puts n
}
end
This strikes me as a terrible example. For example, this is
significantly clearer:
def print_numbers()
for n in [1,2,3,4,5,6]:
square, cube = n * n, n * n * n
if square != 25 and cube != 64:
print n
I /can/ see arguments for ruby style blocks in python, but not for
this sort of thing, or lisp style quoted expressions[1]. ie I can see
situations where you have more complex code in real life where they
will definitely simplify things.
[1] This is perhaps more appropriate because '(a b c) is equivalent
to (quote a b c), and quote a b c can be viewed as close to
python's expression lambda: a b c
However, I can also see that in simple situations - such as the
example you post - they will have a tendency to make code
significantly less clear/direct.
I suppose, if I have a choice between something (hard being possible
simple code looking simple) and (hard things being simpler simple
things looking harder), I'd probably personally choose the former.
This is not because I don't like hard things being simple, but because
I think that simple things are more common and making them look harder
is a mistake.
I'm well aware that's opinion however,
Regards,
Michael.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Precision issue in python
A quick solution I came out with, no stirling numbers and had tried to avoid
large integer multiplication as much as possible.
import math
for i in range(int(raw_input())):
n, k, l = [int(i) for i in raw_input().split()]
e = sum(math.log10(i) for i in range(1, n+1))
frac_e = e - math.floor(e)
a = str(10**frac_e * 10**(k - 1)).split('.')[0]
b = 1
for i in range(n, 0, -1):
b = (b * i) % 10**l
lb = len(str(b))
if lb l:
b = str(b) + '0'* (l - lb)
print a, b
~l0nwlf
On Sat, Feb 20, 2010 at 6:14 PM, Mark Dickinson dicki...@gmail.com wrote:
On Feb 20, 11:17 am, mukesh tiwari mukeshtiwari.ii...@gmail.com
wrote:
Hello everyone. I think it is related to the precision with double
arithmetic so i posted here.I am trying with this problem (
https://www.spoj.pl/problems/CALCULAT) and the problem say that Note :
for
all test cases whose N=100, its K=15. I know precision of doubles
in c is 16 digits. Could some one please help me with this precision
issue.I used stirling (http://en.wikipedia.org/wiki/
Stirling's_approximation) to calculate the first k digits of N.
Thank you.
If I understand you correctly, you're trying to compute the first k
digits in the decimal expansion of N!, with bounds of k = 15 and 100
= N 10**8. Is that right?
Python's floats simply don't give you enough precision for this:
you'd need to find the fractional part of log10(N!) with = 15 digits
of precision. But the integral part needs ~ log10(log10(N!)) digits,
so you end up needing to compute log10(N!) with at least 15 +
log10(log10(N!)) ~ 15 + log10(N) + log10(log10(N)) digits (plus a few
extra digits for safety); so that's at least 25 digits needed for N
close to 10**8.
The decimal module would get you the results you need (are you allowed
imports?). Or you could roll your own log implementation based on
integer arithmetic.
--
Mark
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: Looking for crossfold validation code
Following is the code I use. I got it from web, but forgot the link. def k_fold_cross_validation(X, K, randomise = False): Generates K (training, validation) pairs from the items in X. Each pair is a partition of X, where validation is an iterable of length len(X)/K. So each training iterable is of length (K-1)*len(X)/K. If randomise is true, a copy of X is shuffled before partitioning, otherwise its order is preserved in training and validation. if randomise: from random import shuffle; X=list(X); shuffle(X) for k in xrange(K): training = [x for i, x in enumerate(X) if i % K != k] validation = [x for i, x in enumerate(X) if i % K == k] yield training, validation Cheers, dksr On Feb 20, 1:15 am, Mark Livingstone livingstonem...@gmail.com wrote: Hello, I am doing research as part of a Uni research Scholarship into using data compression for classification. What I am looking for is python code to handle the crossfold validation side of things for me - that will take my testing / training corpus and create the testing / training files after asking me for number of folds and number of times (or maybe allow me to enter a random seed or offset instead of times.) I could then either hook my classifier into the program or use it in a separate step. Probably not very hard to write, but why reinvent the wheel ;-) Thanks in advance, MarkL -- http://mail.python.org/mailman/listinfo/python-list
Re: Precision issue in python
On Feb 20, 5:44 pm, Mark Dickinson dicki...@gmail.com wrote: On Feb 20, 11:17 am, mukesh tiwari mukeshtiwari.ii...@gmail.com wrote: Hello everyone. I think it is related to the precision with double arithmetic so i posted here.I am trying with this problem (https://www.spoj.pl/problems/CALCULAT) and the problem say that Note : for all test cases whose N=100, its K=15. I know precision of doubles in c is 16 digits. Could some one please help me with this precision issue.I used stirling (http://en.wikipedia.org/wiki/ Stirling's_approximation) to calculate the first k digits of N. Thank you. If I understand you correctly, you're trying to compute the first k digits in the decimal expansion of N!, with bounds of k = 15 and 100 = N 10**8. Is that right? Python's floats simply don't give you enough precision for this: you'd need to find the fractional part of log10(N!) with = 15 digits of precision. But the integral part needs ~ log10(log10(N!)) digits, so you end up needing to compute log10(N!) with at least 15 + log10(log10(N!)) ~ 15 + log10(N) + log10(log10(N)) digits (plus a few extra digits for safety); so that's at least 25 digits needed for N close to 10**8. The decimal module would get you the results you need (are you allowed imports?). Or you could roll your own log implementation based on integer arithmetic. -- Mark Yes i am trying to first k digits of N!.I will try your method. Thank you -- http://mail.python.org/mailman/listinfo/python-list
Re: Few questions on SOAP
Thanks every one for commenting. I guess I misspoke. I meant to say that the group is not necessarily the best for parts of this question, so Subhabrata might not get as enthusiastic responses as in some other lists (which i don't recollect at the moment, sorry). I didn't want to convey the sense that his question is not welcome, rather that it might not get a lot of answers. Thanks Brendon and Steve for the support, and thanks Mark for correcting my slip. Regards, k -- http://mail.python.org/mailman/listinfo/python-list
Re: Precision issue in python
On Feb 20, 8:13 pm, mukesh tiwari mukeshtiwari.ii...@gmail.com wrote: On Feb 20, 5:44 pm, Mark Dickinson dicki...@gmail.com wrote: On Feb 20, 11:17 am, mukesh tiwari mukeshtiwari.ii...@gmail.com wrote: Hello everyone. I think it is related to the precision with double arithmetic so i posted here.I am trying with this problem (https://www.spoj.pl/problems/CALCULAT) and the problem say that Note : for all test cases whose N=100, its K=15. I know precision of doubles in c is 16 digits. Could some one please help me with this precision issue.I used stirling (http://en.wikipedia.org/wiki/ Stirling's_approximation) to calculate the first k digits of N. Thank you. If I understand you correctly, you're trying to compute the first k digits in the decimal expansion of N!, with bounds of k = 15 and 100 = N 10**8. Is that right? Python's floats simply don't give you enough precision for this: you'd need to find the fractional part of log10(N!) with = 15 digits of precision. But the integral part needs ~ log10(log10(N!)) digits, so you end up needing to compute log10(N!) with at least 15 + log10(log10(N!)) ~ 15 + log10(N) + log10(log10(N)) digits (plus a few extra digits for safety); so that's at least 25 digits needed for N close to 10**8. The decimal module would get you the results you need (are you allowed imports?). Or you could roll your own log implementation based on integer arithmetic. -- Mark Yes i am trying to first k digits of N!.I will try your method. Thank you I am out of luck.I put d_2=d_1-int(d_1)+25 instead of d_2=d_1- int(d_1)+K-1 but i am getting WA. The decimal module would get you the results you need (are you allowed imports?). Or you could roll your own log implementation based on integer arithmetic. I don't know if is possible to import this decimal module but kindly tell me.Also a bit about log implementation Thank you -- http://mail.python.org/mailman/listinfo/python-list
Re: Precision issue in python
I don't know if is possible to import this decimal module but kindly tell me.Also a bit about log implementation Why don't you read about decimal module (there is log too in it) and try writing your approach here in case it does not work? Or you insist someone to rewrite your code using decimal module ? :-/ ~l0nwlf -- http://mail.python.org/mailman/listinfo/python-list
Pure virtual functions in Python?
Hello How can I do something similar to pure virtual functions in C++ ? Let us consider this: class C1: # Pure virtual def cb(self, param1, param2): This is a callback @param param1: ... @param param2: ... raise NotImplementedError, Implement me # Implementation w/o a 'cb', thus 'cb' should not be used class C2(C1): def __init__(self): pass # Implementation w/ 'cb', thus 'cb' can be used class C3(C1): def __init__(self): pass def cb(self, param1, param2): print i am c3 cb # Dispatcher function that calls 'cb' only if 'cb' is implemented in child classes def dispatcher(c): if hasattr(c, 'cb'): c.cb(Hello, World) dispatcher(C2()) dispatcher(C3()) What I want is the ability to have the dispatcher() not to call 'cb' if it was not implemented in one of the child classes. Please advise. -- http://mail.python.org/mailman/listinfo/python-list
finding element by tag in xml
Hi
I am trying to search an element by tag and new in reading a xml file
(in python). I coded this , but it did not work
--
'''This is to detect the first element and print out all that element
by tag'''
from xml.dom.minidom import parse
from xml.etree.ElementTree import*
file1=book.xml
tmptree=ElementTree()
tmptree.parse(file1)
items=root.getiterator()
dom = parse(file1)
#Find tag names
for node in items :
if node.tag == 'author': #Get tag
print dom.getElementsByTagName ('book') #Error 1
-'
2 Questions
1. Why can't I use dom.getElementsByTagName('book') in #Error 1? How
do i print the elements ?
Error- AttributeError: ElementTree instance has no attribute
'getElementsByTagName'
Thanks
John
--
http://mail.python.org/mailman/listinfo/python-list
Re: Upgrading Py2exe App
On Feb 19, 4:32 pm, Ryan Kelly r...@rfk.id.au wrote: On Fri, 2010-02-19 at 11:08 -0800, T wrote: On Feb 18, 7:19 pm, Ryan Kelly r...@rfk.id.au wrote: On Thu, 2010-02-18 at 07:46 -0800, T wrote: I have a Python app which I converted to an EXE (all files separate; single EXE didn't work properly) via py2exe - I plan on distributing this and would like the ability to remotely upgrade the program (for example, via HTTP/HTTPS). Looks like it's not just the EXE that I will need need to replace (DLLs, the library.zip, etc.). What would be the best way to go about doing this? I've been working on an auto-update framework for my own frozen apps, you might find it useful: http://pypi.python.org/pypi/esky Docs are a little scarce at the moment, the next release will hopefully come with a short tutorial (as well as support for cx_freeze and maybe py2app, depending on how adventurous I'm feeling). Thanks Ryan..this looks like it could be what I'm looking for, but I'm still a bit unsure of how exactly how it works. Do you happen to have an idea approx when the next release w/ tutorial will be out? If I punt on the py2app support, I should be able to get it done in the next 3-4 days. I'll send a quick email to python-list when it's ready. Here's a rough roadmap of where the project is heading: v0.4.0: cx_freeze support and a tutorial [the next 3-4 days] v0.4.1: py2app support [the next 2-3 weeks] v0.5.0: differential updates using bsdiff [next few months] Cheers, Ryan -- Ryan Kellyhttp://www.rfk.id.au | This message is digitally signed. Please visit r...@rfk.id.au | http://www.rfk.id.au/ramblings/gpg/for details signature.asc 1KViewDownload Excellent - I look forward to giving it a try. Thanks again! -- http://mail.python.org/mailman/listinfo/python-list
Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.
On Feb 20, 6:13 am, Michael Sparks spark...@gmail.com wrote:
On Feb 18, 4:15 pm, Steve Howell showel...@yahoo.com wrote:
...
def print_numbers()
[1, 2, 3, 4, 5, 6].map { |n|
[n * n, n * n * n]
}.reject { |square, cube|
square == 25 || cube == 64
}.map { |square, cube|
cube
}.each { |n|
puts n
}
end
This strikes me as a terrible example. For example, this is
significantly clearer:
def print_numbers()
for n in [1,2,3,4,5,6]:
square, cube = n * n, n * n * n
if square != 25 and cube != 64:
print n
This is not an exact translation. My example prints the cubes. It is
my fault for using n as the parameter in the last block. I would
rename the parameter to cube.
I /can/ see arguments for ruby style blocks in python, but not for
this sort of thing, or lisp style quoted expressions[1]. ie I can see
situations where you have more complex code in real life where they
will definitely simplify things.
[1] This is perhaps more appropriate because '(a b c) is equivalent
to (quote a b c), and quote a b c can be viewed as close to
python's expression lambda: a b c
However, I can also see that in simple situations - such as the
example you post - they will have a tendency to make code
significantly less clear/direct.
I suppose, if I have a choice between something (hard being possible
simple code looking simple) and (hard things being simpler simple
things looking harder), I'd probably personally choose the former.
This is not because I don't like hard things being simple, but because
I think that simple things are more common and making them look harder
is a mistake.
I agree with much of what you are saying. The example is indeed
terribly contrived.
I'm not sure I agree that there is anything unclear or undirect about
the Ruby, though. I've been fairly immersed in Ruby code, so maybe
it's been warping my brain, but once you get over the unfamiliarity of
the syntax, you see that there's actually a rhythm to the code.
Setting aside punctuation and parameter lists, the code clearly
expresses the transformations and actions in the natural order that
you'd do them:
LIST map
expression
reject
criteria
map
expression
each
statement
In English, for the list elements, map them to tuples of squares and
cubes, reject the oddballs, take the cube, and print it out.
[1, 2, 3, 4, 5, 6].map { |n|
[n * n, n * n * n]
}.reject { |square, cube|
square == 25 || cube == 64
}.map { |square, cube|
cube
}.each { |cube|
puts cube
}
For such a small problem, I agree it's verbose. But it's also
completely flat--you don't need to use an if statement to express
the concept of rejection.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Pure virtual functions in Python?
lallous wrote: Hello How can I do something similar to pure virtual functions in C++ ? See, for example http://code.activestate.com/recipes/266468/ Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Pure virtual functions in Python?
Am 20.02.10 17:12, schrieb lallous: Hello How can I do something similar to pure virtual functions in C++ ? Let us consider this: class C1: # Pure virtual def cb(self, param1, param2): This is a callback @param param1: ... @param param2: ... raise NotImplementedError, Implement me # Implementation w/o a 'cb', thus 'cb' should not be used class C2(C1): def __init__(self): pass # Implementation w/ 'cb', thus 'cb' can be used class C3(C1): def __init__(self): pass def cb(self, param1, param2): print i am c3 cb # Dispatcher function that calls 'cb' only if 'cb' is implemented in child classes def dispatcher(c): if hasattr(c, 'cb'): c.cb(Hello, World) dispatcher(C2()) dispatcher(C3()) What I want is the ability to have the dispatcher() not to call 'cb' if it was not implemented in one of the child classes. Please advise. There is nothing more beyond that what you already did. You can raise a NotImplementedError for classes that don't implement the method. That's it. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Pure virtual functions in Python?
class C1: # Pure virtual def cb(self, param1, param2): This is a callback @param param1: ... @param param2: ... raise NotImplementedError, Implement me # Dispatcher function that calls 'cb' only if 'cb' is implemented in child classes def dispatcher(c): if hasattr(c, 'cb'): c.cb(Hello, World) dispatcher(C2()) dispatcher(C3()) What I want is the ability to have the dispatcher() not to call 'cb' if it was not implemented in one of the child classes. Please advise. There is nothing more beyond that what you already did. You can raise a NotImplementedError for classes that don't implement the method. That's it. That's not true. Currently, the hasattr() call would report that cb is available, when it is actually not implemented. It would be possible to do something like if hasattr(c, 'cb') and not is_pure(c.cb): c.cb(Hello, World) is_pure could, for example, look at a function attribute of the callback. You'd write something like @pure_virtual def cb(self, param1, param2): not_implemented Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Pure virtual functions in Python?
Sorry, I totally mis-read the OP, too tired. You are right of course. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Pure virtual functions in Python?
On Saturday 20 February 2010 11:46:42 Diez B. Roggisch wrote: Am 20.02.10 17:12, schrieb lallous: Hello How can I do something similar to pure virtual functions in C++ ? Let us consider this: class C1: # Pure virtual def cb(self, param1, param2): This is a callback @param param1: ... @param param2: ... raise NotImplementedError, Implement me # Implementation w/o a 'cb', thus 'cb' should not be used class C2(C1): def __init__(self): pass # Implementation w/ 'cb', thus 'cb' can be used class C3(C1): def __init__(self): pass def cb(self, param1, param2): print i am c3 cb # Dispatcher function that calls 'cb' only if 'cb' is implemented in child classes def dispatcher(c): if hasattr(c, 'cb'): c.cb(Hello, World) dispatcher(C2()) dispatcher(C3()) What I want is the ability to have the dispatcher() not to call 'cb' if it was not implemented in one of the child classes. Please advise. There is nothing more beyond that what you already did. You can raise a NotImplementedError for classes that don't implement the method. That's it. Diez Perhaps you could use an easier-to-ask-forgiveness-than-permission idiom? def dispatcher(c): try: c.cb(Hello, World) except NotImplementedError: pass Rami Chowdhury Passion is inversely proportional to the amount of real information available. -- Benford's Law of Controversy 408-597-7068 (US) / 07875-841-046 (UK) / 01819-245544 (BD) -- http://mail.python.org/mailman/listinfo/python-list
Re: Precision issue in python
On Feb 20, 3:37 pm, mukesh tiwari mukeshtiwari.ii...@gmail.com
wrote:
I don't know if is possible to import this decimal module but kindly
tell me.Also a bit about log implementation
The decimal module is part of the standard library; I don't know what
the rules are for SPOJ, but you're already importing the math module,
so at least *some* imports are obviously permitted. Here's a quick
demonstration of how you might use the decimal module: you can
probably find ways to tweak it for speed and accuracy.
from decimal import Decimal as D
from decimal import getcontext
getcontext().prec = 100 # working precision = 100 digits
pi =
D('3.14159265358979323846264338327950288419716939937510582097494459'
'2307816406286208998628034825342117067982148086513282306647093844')
half = D('0.5')
log = D.ln
def logfac(x):
Approximation to log(x!), using first few terms of Stirling's
series.
x = D(x)
return log(2*pi)/2 + (x + half)*log(x) - x + \
1/(12*x) - 1/(360*x**3) + 1/(1260*x**5)
def fac_first_digits(n, k):
Get first k decimal digits of n!.
log10_nfac = logfac(n)/log(D(10))
frac = log10_nfac - int(log10_nfac)
return int(10**(frac - 1 + k))
With the above code I get, for example (with Python 2.6):
fac_first_digits(12345, 15)
344364246918678
from math import factorial
int(str(factorial(12345))[:15])
344364246918678
For small n, you'll need more terms of Stirling's series than I've
used above. And there's always a small chance that the intermediate
errors involved in the computation might change those first 15
digits; with a working precision of 100 and lots of terms of
Stirling's series it's a *very* small chance, but it's there. If you
want something 100% watertight, you'd need to compute strict upper
and lower bounds for the true result, and increase precision until
those bounds match sufficiently far that you can be sure of the first
k digits being correct.
--
Mark
--
http://mail.python.org/mailman/listinfo/python-list
Building a dict from a tuple of tuples
Hello everyone!
I have a tuple of tuples, coming from an Excel range, such as this:
((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
I need to build a dictionary that has, as key, the row and column
header.
For example:
d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }
As you can see, if the value in the matrix is None, no key has to be
added to the dictionary.
Of course, my tuple of tuples is a lot bigger.
How can I possibly do this?
Thank you
--
http://mail.python.org/mailman/listinfo/python-list
Re: Can't Access ANY url from python (errno 61)
On 02/20/10 19:36, MattB wrote: On Feb 20, 2:02 am, Lie Ryan lie.1...@gmail.com wrote: On 02/20/10 13:32, MattB wrote: I'm using the network in my own apartment. Not the campus's. Moreover, my mac's MAC address is different from the MAC address shown by my router, but as I said I'm also blocked when using my friend's wireless router at his apartment. So it must be my mac's MAC, and not the router's MAC, that's being blocked, right? But ALSO -- is it my ISP that's blocking the mac's MAC (and not the school), since I can't raise ANY url's from python when I'm on wireless? MAC or IP blocking can't be the reason, as the OP stated, he can use Firefox just fine. Can you access, say,http://www.google.comfrom urllib or mechanize? If you can't access *any website* using urllib/mechanize but you can with a browser and you're on a unix-based machine, you probably have the same problem as I used to have. Check whether you used the same hostname in /etc/conf.d/hostname and /etc/hosts (or wherever your distro saves its hostname configurations, I use Gentoo); after editing those files reboot (there are ways to avoid reboot, but rebooting guarantees the conf file is reread). Check the hostname by running this python script: import socket hn = socket.gethostname() print hn print socket.gethostbyname(hn) # should be 127.0.0.1 Lie, Wow. Strangely, that script returned 192.168.1.106. However, in Snow Leopard's airport settings, if I click on 'advanced' and then 'proxies', the default proxy for 'http' is 127.0.0.1: (and in these settings, the 'use proxy for http' is checked). I just tried checking the unix files you mentioned. In etc/hosts, the following info is displayed: ## # Host Database # # localhost is used to configure the loopback interface # when the system is booting. Do not change this entry. ## 127.0.0.1 localhost 255.255.255.255 broadcasthost ::1 localhost fe80::1%lo0 localhost Not sure if these are what I'm looking for -- I'm new to unix so I may need a bit more hand-holding here. Try the alternatives in this site: http://movealong.org/hostname.html Also found a file called ntp-restrict.conf, containing: # Access restrictions documented in ntp.conf(5) and # http://support.ntp.org/bin/view/Support/AccessRestrictions # Limit network machines to time queries only restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery # localhost is unrestricted restrict 127.0.0.1 restrict -6 ::1 includefile /private/etc/ntp.conf Not sure if ntp-restrict.conf is a red herring here since its name implies it's something for NTP (Network Time Protocol, used for time synchronization) -- http://mail.python.org/mailman/listinfo/python-list
Re: the mystery of dirname()
Shashwat Anand wrote:
basically I infer that : dirname = path - basename, like for path =
'//x', basename = x, hence dirname = '//'
[snip]
Basically, os.path.dirname() should return the directory name, which
means dropping everything after the last slash, and also the last slash.
However, there's the special case where the name is in the root
directory, and you want to retain the slash(es).
When building a path (os.path.join()) you want to join the parts with a
single slash between them, but there's the special case when a part is
the root directory, and you don't want to add a slash between them, eg
os.part.join('/x', 'y') should return '/x/y', but os.part.join('/', 'x')
should return '/x'.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Precision issue in python
On Sat, Feb 20, 2010 at 2:42 PM, Shashwat Anand
anand.shash...@gmail.com wrote:
A quick solution I came out with, no stirling numbers and had tried to avoid
large integer multiplication as much as possible.
import math
for i in range(int(raw_input())):
n, k, l = [int(i) for i in raw_input().split()]
e = sum(math.log10(i) for i in range(1, n+1))
frac_e = e - math.floor(e)
This isn't going to give you enough accuracy when n gets large (and
the original problem seems to allow n to be as large as 10**8), for
the reasons I described earlier---that is, Python floats only give you
around 16 decimal digits of precision; your 'e' value will already
have used up some of those 16 digits before the point, leaving you
fewer than 16 digits of precision after the point, so the absolute
error in frac_e will likely be larger than 1e-15. That's not good
enough for getting the first 15 digits of 10**frac_e accurately. For
large n, you'll also be accumulating significant error in the sum.
a = str(10**frac_e * 10**(k - 1)).split('.')[0]
The str(...).split('.') here doesn't do a good job of extracting the
integer part when its argument is = 1e12, since Python produces a
result in scientific notation. I think you're going to get strange
results when k = 13.
--
Mark
--
http://mail.python.org/mailman/listinfo/python-list
Re: Building a dict from a tuple of tuples
vsoler wrote:
Hello everyone!
I have a tuple of tuples, coming from an Excel range, such as this:
((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
I need to build a dictionary that has, as key, the row and column
header.
For example:
d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }
As you can see, if the value in the matrix is None, no key has to be
added to the dictionary.
Of course, my tuple of tuples is a lot bigger.
How can I possibly do this?
Thank you
Does this help?
matrix = ((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
for row in matrix[1 : ]:
for col, val in zip(matrix[0][1 : ], row[1 : ]):
print row[0], col, val
--
http://mail.python.org/mailman/listinfo/python-list
Re: Can't Access ANY url from python (errno 61)
On 02/20/10 00:20, MattB wrote: cut Also, based on Martin's comment, I just wanted to make you all aware that I intend no misuse, but rather am just trying to learn, as I'm a programming noob. cut It wasn't my intention to imply that, rather the opposite, that if some BOFH would see your action as misuse (some admins are pretty trigger happy), you would expect them to contact you directly (I would). Even though you are on a wireless there are ways to track you down. For example I would search my log to see if I can make an association between a login to one of my servers and your mac address. BTW, I always followed the philosophy that learning is much more fun if you can brake/blow something up. Thus on my networks all students had a lot of freedom to do whatever they think was appropriate but they where aware that every action on my network was monitored and logged. -- mph -- http://mail.python.org/mailman/listinfo/python-list
Re: the mystery of dirname()
got it. thanks. :)
On Sat, Feb 20, 2010 at 11:19 PM, MRAB pyt...@mrabarnett.plus.com wrote:
Shashwat Anand wrote:
basically I infer that : dirname = path - basename, like for path =
'//x', basename = x, hence dirname = '//'
[snip]
Basically, os.path.dirname() should return the directory name, which
means dropping everything after the last slash, and also the last slash.
However, there's the special case where the name is in the root
directory, and you want to retain the slash(es).
When building a path (os.path.join()) you want to join the parts with a
single slash between them, but there's the special case when a part is
the root directory, and you don't want to add a slash between them, eg
os.part.join('/x', 'y') should return '/x/y', but os.part.join('/', 'x')
should return '/x'.
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: Pure virtual functions in Python?
lallous wrote: How can I do something similar to pure virtual functions in C++ ? http://docs.python.org/library/abc.html#abc.abstractmethod Peter -- http://mail.python.org/mailman/listinfo/python-list
Re: Precision issue in python
@Mark,
The str(...).split('.') here doesn't do a good job of extracting the
integer part when its argument is = 1e12, since Python produces a
result in scientific notation. I think you're going to get strange
results when k = 13.
Yeah, you were correct. I tested it for k = 13, and there were issues i.e.
only one character is printed.
Shashwat-Anands-MacBook-Pro:Desktop l0nwlf$ python CALCULAT.py
3
100 12 12
826393051664
100 13 13
8 0
100 15 4
9
The logic I tried was : for
alpha = n!
log(alpha) = log(n!)
= log(n) + log(n-1) + .. log(2) + log(1)
= e [where e = log(n) + + log(1)]
frac_e = {e} (fractional part of e, like {1.23} = .23)
now last k digits of alpha = 10**frac_e * 10**(k - 1)
As I can see, the method is mathematically sound, and there is precision
issues. Time for code modification.
~l0nwlf
--
http://mail.python.org/mailman/listinfo/python-list
Compiling and running 32-bit Python on 64-bit server?
Hi, Some server-side Python applications are limited by memory usage (hint: Zope), because Python effective uses processes and not threads for multiprocessing. This is especially true for 64-bit platforms, since Python programs are all about references and objects and 64-bit effectively doubles reference size. Some benchmarks 32-bit vs. 64-bit were discussed here: http://jstahl.org/archives/2010/01/2...ks-python-2-6/ How one could create 32-bit Python run-time enviroment, preferable virtualenv, on 64-bit Linux (VPS), reducing memory usage? This environment could actually beat 64-bit in performance, due to better memory cache use. I assume this involves having lib32 libs and compiling Python with some magical switches. Cheers, Mikko -- http://mail.python.org/mailman/listinfo/python-list
Re: Building a dict from a tuple of tuples
On Feb 20, 7:00 pm, MRAB pyt...@mrabarnett.plus.com wrote:
vsoler wrote:
Hello everyone!
I have a tuple of tuples, coming from an Excel range, such as this:
((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
I need to build a dictionary that has, as key, the row and column
header.
For example:
d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }
As you can see, if the value in the matrix is None, no key has to be
added to the dictionary.
Of course, my tuple of tuples is a lot bigger.
How can I possibly do this?
Thank you
Does this help?
matrix = ((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
for row in matrix[1 : ]:
for col, val in zip(matrix[0][1 : ], row[1 : ]):
print row[0], col, val
and the dictionary?
it is the ultimate goal of what I am intending...
Thank you
--
http://mail.python.org/mailman/listinfo/python-list
Re: os.pipe() + os.fork()
Sebastian Noack wrote: I have figured out that, you have to close the writing end in the child process, which is reading from the pipe. Otherwise the underlying pipe is not going to be closed when the parent process is closing its writing end. This has nothing to do with Python itself. I have tried plain C and there it is the same behaviour. Regards Sebastian Noack Correct. The fork creates two processes with references to the read and write ends of the pipe. Both parent and child processes should close the ends they are not using. Here's a thought: Consider the subprocess module. It can do the fork and any necessary pipes and can do so in an OS independent way. It might make you life much easier. Gary Herron -- http://mail.python.org/mailman/listinfo/python-list
Re: Compiling and running 32-bit Python on 64-bit server?
How one could create 32-bit Python run-time enviroment, preferable virtualenv, on 64-bit Linux (VPS), reducing memory usage? I'd install a 32-bit Linux on the hardware, and install a bigmem kernel if it has more than 3GB of main memory. I assume this involves having lib32 libs and compiling Python with some magical switches. The precise set of packages that you will need depends on the specific Linux distribution. Check whether gcc -m32 can build a hello-world program. Then, set CC to gcc -m32, and follow the build instructions in Python's README file. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
if not global -- then what?
Hi, newbie here. I've read on using the 'global' keyword being discouraged; then what is the preferred way to have something, for example a class containing program settings, accessible from everywhere, in a program spanning multiple files? -- http://mail.python.org/mailman/listinfo/python-list
Re: Building a dict from a tuple of tuples
vsoler wrote:
On Feb 20, 7:00 pm, MRAB pyt...@mrabarnett.plus.com wrote:
vsoler wrote:
Hello everyone!
I have a tuple of tuples, coming from an Excel range, such as this:
((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
I need to build a dictionary that has, as key, the row and column
header.
For example:
d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }
As you can see, if the value in the matrix is None, no key has to be
added to the dictionary.
Of course, my tuple of tuples is a lot bigger.
How can I possibly do this?
Thank you
Does this help?
matrix = ((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
for row in matrix[1 : ]:
for col, val in zip(matrix[0][1 : ], row[1 : ]):
print row[0], col, val
and the dictionary?
it is the ultimate goal of what I am intending...
Thank you
The difficult bit is working out how to produce the keys and values for
the dict from the tuple of tuples, and I've shown you that. The rest is
straightforward.
--
http://mail.python.org/mailman/listinfo/python-list
Re: if not global -- then what?
egasimus wrote: Hi, newbie here. I've read on using the 'global' keyword being discouraged; then what is the preferred way to have something, for example a class containing program settings, accessible from everywhere, in a program spanning multiple files? Python's 'global' keyword is global in a file (module), not global across multiple files, so it would have helped you anyway. If you want program settings accessible across multiple modules then put the settings in a module and import that module in any other module that needs to access them. -- http://mail.python.org/mailman/listinfo/python-list
Re: if not global -- then what?
egasimus wrote: Hi, newbie here. I've read on using the 'global' keyword being discouraged; then what is the preferred way to have something, for example a class containing program settings, accessible from everywhere, in a program spanning multiple files? Define your global in a module (a .py file) and import that wherever you need it: mod.py: gvar = 123 # A Global (like) value main.py: import mod ... mod.gvar ...# To access the value ... mod.gvar = 456 ... #To change the value Using the global keyword has nothing to do with this. And in fact Python does not even have global variables in the sense you may be thinking. If you wish to provide a function in mode.py that sets the value then this would fail mod.py: gvar=123 def SetGvar(v): gvar = v because the gvar inside the function is local to that function. This is the proper place to use the global keyword. mode.py: gvar = 123 def SetGvar(v): global gvar # Makes gvar refer to the (global) module level gvar gvar = v Gary Herron -- http://mail.python.org/mailman/listinfo/python-list
Re: if not global -- then what?
On Sat, Feb 20, 2010 at 8:25 PM, egasimus fallenbl...@gmail.com wrote: Hi, newbie here. I've read on using the 'global' keyword being discouraged; then what is the preferred way to have something, for example a class containing program settings, accessible from everywhere, in a program spanning multiple files? -- http://mail.python.org/mailman/listinfo/python-list There is probably a smarter way but I would recommend passing a settings object around. -- http://mail.python.org/mailman/listinfo/python-list
Re: Pure virtual functions in Python?
On Sat, 20 Feb 2010 08:12:01 -0800, lallous wrote: How can I do something similar to pure virtual functions in C++ ? From what you want, it seems like you want cb() to not be called if it isn't implemented in the derived class; this isn't really what pure virtual functions in C++ do - pure virtual functions enforce, at compile time, that the derived class implements the method. If you have a situation when you want to either call a derived class's version of cb(), or do nothing, can you not just have an implementation of cb() in the base class that does nothing, i.e. class C1(object): def cb(self, param1, param2): pass -- http://mail.python.org/mailman/listinfo/python-list
How would I do a continuous write over a pipe in the following code...
Given the following #!/usr/bin/python import subprocess as s broadcast = s.Popen(echo test | wall, shell=True,stdout=s.PIPE) out = broadcast.stdout while 1: out broadcast.wait() broadcast.stdout.close() The code only executes once. What I want to do is be able to continuously write over the pipe once it is open. I could put s.Popen() inside the while loop, but that seems a bit too messy. So is there some way to just open the pipe once, and once it is open, just continuously write over it vs just opening and closing the pipe every time? -- http://mail.python.org/mailman/listinfo/python-list
Re: Building a dict from a tuple of tuples
On Feb 20, 8:54 pm, MRAB pyt...@mrabarnett.plus.com wrote:
vsoler wrote:
On Feb 20, 7:00 pm, MRAB pyt...@mrabarnett.plus.com wrote:
vsoler wrote:
Hello everyone!
I have a tuple of tuples, coming from an Excel range, such as this:
((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
I need to build a dictionary that has, as key, the row and column
header.
For example:
d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }
As you can see, if the value in the matrix is None, no key has to be
added to the dictionary.
Of course, my tuple of tuples is a lot bigger.
How can I possibly do this?
Thank you
Does this help?
matrix = ((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
for row in matrix[1 : ]:
for col, val in zip(matrix[0][1 : ], row[1 : ]):
print row[0], col, val
and the dictionary?
it is the ultimate goal of what I am intending...
Thank you
The difficult bit is working out how to produce the keys and values for
the dict from the tuple of tuples, and I've shown you that. The rest is
straightforward.
I'll try. Thank you very much MRAB
--
http://mail.python.org/mailman/listinfo/python-list
Re: finding element by tag in xml
On Feb 20, 11:27 am, sWrath swrath swr...@gmail.com wrote:
2 Questions
1. Why can't I use dom.getElementsByTagName('book') in #Error 1? How
do i print the elements ?
Error- AttributeError: ElementTree instance has no attribute
'getElementsByTagName'
I only see one question here.
I think the error is asserting that your instance of the ET class does
not have such a fcn. I also do not see it in the ET documentation.
Perhaps you are looking for the XPath support at
http://effbot.org/zone/element-xpath.htm
? Or perhaps you want to make your document dom representation via a
different library?
Jim
--
http://mail.python.org/mailman/listinfo/python-list
Re: finding element by tag in xml
On Sat, Feb 20, 2010 at 9:27 AM, sWrath swrath swr...@gmail.com wrote:
from xml.dom.minidom import parse
from xml.etree.ElementTree import*
file1=book.xml
tmptree=ElementTree()
tmptree.parse(file1)
items=root.getiterator()
dom = parse(file1)
#Find tag names
for node in items :
if node.tag == 'author': #Get tag
print dom.getElementsByTagName ('book') #Error 1
You are mixing two different types of xml parsers, either user minidom
for node in dom.getElementsByTagName('author'):
print node.getElementsByTagName('book')
or use etree
for el in items:
if el.tag == 'author':
print el.find('book')
There are a few differences you need to note, the getElementsByTagName
always returns a list even if there is only 1 element and find only
returns the first element found no matter what. If you want to print
all books associated with an author you would need to do something
like
for author in tmptree.getiterator('author'):
for book in author.getiterator('book'):
print book
--
http://mail.python.org/mailman/listinfo/python-list
Re: Pure virtual functions in Python?
lallous elias.bachaal...@gmail.com writes: Hello How can I do something similar to pure virtual functions in C++ ? Let us consider this: class C1: # Pure virtual def cb(self, param1, param2): This is a callback @param param1: ... @param param2: ... raise NotImplementedError, Implement me Why define it if it is virtual? # Implementation w/o a 'cb', thus 'cb' should not be used class C2(C1): def __init__(self): pass # Implementation w/ 'cb', thus 'cb' can be used class C3(C1): def __init__(self): pass def cb(self, param1, param2): print i am c3 cb # Dispatcher function that calls 'cb' only if 'cb' is implemented in child classes def dispatcher(c): if hasattr(c, 'cb'): c.cb(Hello, World) dispatcher(C2()) dispatcher(C3()) What I want is the ability to have the dispatcher() not to call 'cb' if it was not implemented in one of the child classes. If you don't define cb in the parent class, it'll work. -- Arnaud -- http://mail.python.org/mailman/listinfo/python-list
Reading a large bz2 textfile exits early
Hello, i am trying to read a large bz2 compressed textfile using the bz2 module. The file is 1717362770 lines long and 8GB large. Using this code source_file = bz2.BZ2File(file, r) for line in source_file: print line.strip() print Exiting print I used file: + file the loop exits cleanly after 4311 lines in midline and the prints are executed. This happened on two different boxes runnig different brands of linux. Is there something i miss or should be done differently? Thank you. Regards, Norman -- http://mail.python.org/mailman/listinfo/python-list
/usr/bin/ld: cannot find -lz on Cent OS - Python 2.4
Been trying to fix this issue for over 6 hours now. It's doin my head in, any one know whats going on here. ==START== python setup.py build running build running build_py copying MySQLdb/release.py - build/lib.linux-x86_64-2.4/MySQLdb running build_ext building '_mysql' extension gcc -pthread -shared build/temp.linux-x86_64-2.4/_mysql.o -L/usr/lib64/ mysql -L/usr/lib64 -lmysqlclient_r -lz -lpthread -lcrypt -lnsl -lm - lpthread -lssl -lcrypto -o build/lib.linux-x86_64-2.4/_mysql.so /usr/bin/ld: skipping incompatible /usr/lib/libz.so when searching for -lz /usr/bin/ld: skipping incompatible /usr/lib/libz.a when searching for - lz /usr/bin/ld: cannot find -lz collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1 ==END== Am trying to get MySQL-python-1.2.3c1 installed on a CentOS 5 box running Python 2.4 I do not want to upgrade my Python version. I have tried: 1. yum install mysql-python 2. yum install librsync-devel 3. yum install python-devel With all return no errors and a notification that there's nothing to install or update. Please help. -- http://mail.python.org/mailman/listinfo/python-list
Re: calculating a string equation
Arnaud Delobelle wrote:
Astan Chee astan.c...@al.com.au writes:
Hi,
I have some variables in my script that looks like this:
vars = {'var_a':'10','var_b':'4'}
eqat = (var_a/2.0) = var_b
result = (var_a+var_b)/7
What I'm trying to do is to plug in var_a and var_b's values from vars
into eqat and see if eqat returns true or false as well as getting the
value of result if these variables were plugged in. How do I do
this?
I'm also expecting eqat and result to contain various python
mathematical operators like **, and compounded ()'s.
I'm not sure how to convert the equation; if I have to make a bunch of
if-statements or if there is a python function that already does
something like this.
Yes: eval()
vars = {'var_a':10 ,'var_b':4}
eqat = (var_a/2.0) = var_b
result = (var_a+var_b)/7
eval(eqat, vars)
False
Hi,
I now have a slight problem with this. This doesnt seem to work:
vars = {'var a':10 ,'var b':4}
eqat = (var a/2.0) = var b
eval(eqat, vars)
Traceback (most recent call last):
File pyshell#11, line 1, in module
eval(eqat, vars)
File string, line 1
(var a/2.0) = var b
^
SyntaxError: invalid syntax
So eval can't take spaces? is there a way to go around this?
Thanks again for any suggestions
Cheers
Astan
--
http://mail.python.org/mailman/listinfo/python-list
Re: calculating a string equation
On Sat, Feb 20, 2010 at 3:07 PM, Astan Chee astan.c...@al.com.au wrote:
Arnaud Delobelle wrote:
Astan Chee astan.c...@al.com.au writes:
Hi,
I have some variables in my script that looks like this:
vars = {'var_a':'10','var_b':'4'}
eqat = (var_a/2.0) = var_b
result = (var_a+var_b)/7
What I'm trying to do is to plug in var_a and var_b's values from vars
into eqat and see if eqat returns true or false as well as getting the
value of result if these variables were plugged in. How do I do
this?
I'm also expecting eqat and result to contain various python
mathematical operators like **, and compounded ()'s.
I'm not sure how to convert the equation; if I have to make a bunch of
if-statements or if there is a python function that already does
something like this.
Yes: eval()
vars = {'var_a':10 ,'var_b':4}
eqat = (var_a/2.0) = var_b
result = (var_a+var_b)/7
eval(eqat, vars)
False
Hi,
I now have a slight problem with this. This doesnt seem to work:
vars = {'var a':10 ,'var b':4}
eqat = (var a/2.0) = var b
eval(eqat, vars)
Traceback (most recent call last):
File pyshell#11, line 1, in module
eval(eqat, vars)
File string, line 1
(var a/2.0) = var b
^
SyntaxError: invalid syntax
So eval can't take spaces? is there a way to go around this?
eval() requires the string be valid Python code and conform to
Python's syntax. Obviously your string isn't valid Python.
You can either require beforehand that variables not be prefixed with
var as they are currently, or you can get rid of the vars
yourself:
vars = {'a':10 ,'b':4}
equat = equat.replace('var ','')
eval(equat, vars)
but this could cause problems if you have a variable whose names ends
with var; using a regular expression for the removal would fix that.
Cheers,
Chris
--
http://blog.rebertia.com
--
http://mail.python.org/mailman/listinfo/python-list
Is there a way to continue after an exception ?
hello, I would like my program to continue on the next line after an uncaught exception, is that possible ? thanks Stef Mientki -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to continue after an exception ?
On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki stef.mien...@gmail.com wrote: hello, I would like my program to continue on the next line after an uncaught exception, is that possible ? thanks Stef Mientki Yes, you catch the exception and do nothing. -- http://mail.python.org/mailman/listinfo/python-list
Re: MODULE FOR I, P FRAME
DANNY danijel.gv...@gmail.com wrote: If I want to have a MPEG-4/10 coded video and stream it through the network and than have the same video on the client side, what should I use and of course I don't want to have raw MPEG data, because than I couldn't extract the frames to manipulate them. If you want to manipulate the frames (as bitmaps), then you have little choice but to decode the MPEG as you receive it, manipulate the bitmaps, and re-encode it back to MPEG. That's going to take a fair amount of time... -- Tim Roberts, t...@probo.com Providenza Boekelheide, Inc. -- http://mail.python.org/mailman/listinfo/python-list
Re: if not global -- then what?
On Sat, 20 Feb 2010 11:25:46 -0800, egasimus wrote: Hi, newbie here. I've read on using the 'global' keyword being discouraged; then what is the preferred way to have something, for example a class containing program settings, accessible from everywhere, in a program spanning multiple files? Such a thing is no different from keeping many global settings. It's not the global KEYWORD that is discouraged, but the over-use of global variables itself. See, for example: http://c2.com/cgi/wiki?GlobalVariablesAreBad http://weblogs.asp.net/wallen/archive/2003/05/08/6750.aspx In a nutshell: shared variables introduce tight coupling between parts of your code that should be independent. Whether they are truly global, or passed around in a Settings object, makes little or no difference. In both cases, the risks and problems are identical. Let me give a fanciful example: suppose your refrigerator, oven and shower all shared a single global temperature setting instead of having three local temperature settings. This would be a recipe for disaster: every operation to any of the three would need to carefully save the current temperature setting, and there's the constant risk of scalding showers, ruined foods, and undercooked meals. Having three different global temperature settings helps a bit, but that just expands the number of variables that everything has access too. Why do the television and the garage door need access to any temperature setting, let alone all three? Python globals aren't as bad as in some other languages, because global means global to a single module, not global to the entire program, but if you create a Settings object and import it from module to module, you break the encapsulation, *and* you then have the problem of needing to make sure it is initialised properly. http://archive.eiffel.com/doc/manuals/technology/bmarticles/joop/globals.html Global constants also aren't as bad, because the coupling is weaker. Unfortunately Python doesn't have constants, except by convention, so there's always the risk that some badly-behaved function might decide to redefine (say) math.pi to 3.15. Unfortunately, you probably can't get rid of globals altogether, but the tricks are (1) use as few of them as possible, and (2) make them local as quickly as possible. For example, you might have something like this: default_header = My special header # A global for option, argument in command_line: if option == --header: default_header = argument class Printer: def print_page(self, text, header=None): if header is None: header = default_header print header print text I would limit the scope of default_header so that it was local to the printer class: for option, argument in command_line: if option == --header: Printer.default_header = argument class Printer: default_header = My special header def print_page(self, text, header=None): if header is None: header = self.default_header print header print text You might not be able to completely eliminate either globals, or a Settings object, but you should be able to make it much, much smaller. In practice, to make testing easier, I'd split the command line processing into two steps: a function that reads and processes the command line into a single local settings object, then a separate function that applies those settings to the various classes. The function that reads the command line doesn't need to know where the settings eventually end up, and the function that applies the settings doesn't need to know where they come from. All the coupling between settings and classes is in one place, the second function, instead of scattered all over your code. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Can't Access ANY url from python (errno 61)
On Sat, 20 Feb 2010 18:28:16 +, Martin P. Hellwig wrote: On 02/20/10 00:20, MattB wrote: cut Also, based on Martin's comment, I just wanted to make you all aware that I intend no misuse, but rather am just trying to learn, as I'm a programming noob. cut It wasn't my intention to imply that, rather the opposite, that if some BOFH would see your action as misuse (some admins are pretty trigger happy), you would expect them to contact you directly (I would). Even though you are on a wireless there are ways to track you down. For example I would search my log to see if I can make an association between a login to one of my servers and your mac address. This may be what you would do, but in my experience, it is more likely that the admin would be far too busy and/or lazy to try to track you down unless (1) they wanted to give you a warning prior to expulsion, or (2) they needed to know who you were prior to laying charges. In my experience they're far more likely to just hit the problem with a hammer by blocking you as much as is technically possible. BTW, I always followed the philosophy that learning is much more fun if you can brake/blow something up. Thus on my networks all students had a lot of freedom to do whatever they think was appropriate but they where aware that every action on my network was monitored and logged. Unfortunately this is the exception rather than the rule, I think. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: speed question, reading csv using takewhile() and dropwhile()
Thanks for the help, this is considerably faster and easier to read (see below). I changed it to avoid the break and I think it makes it easy to understand. I am checking the conditions each time slows it but it is worth it to me at this time. Thanks again Vincent def read_data_file(filename): reader = csv.reader(open(filename, U),delimiter='\t') data = [] mask = [] outliers = [] modified = [] data_append = data.append mask_append = mask.append outliers_append = outliers.append modified_append = modified.append maskcount = 0 outliercount = 0 modifiedcount = 0 for row in reader: if '[MASKS]' in row: maskcount += 1 if '[OUTLIERS]' in row: outliercount += 1 if '[MODIFIED]' in row: modifiedcount += 1 if not any((maskcount, outliercount, modifiedcount, not row)): data_append(row) elif not any((outliercount, modifiedcount, not row)): mask_append(row) elif not any((modifiedcount, not row)): outliers_append(row) else: if row: modified_append(row) data = data[1:] mask = mask[3:] outliers = outliers[3:] modified = modified[3:] return [data, mask, outliers, modified] *Vincent Davis 720-301-3003 * vinc...@vincentdavis.net my blog http://vincentdavis.net | LinkedInhttp://www.linkedin.com/in/vincentdavis On Fri, Feb 19, 2010 at 4:36 PM, Jonathan Gardner jgard...@jonathangardner.net wrote: On Fri, Feb 19, 2010 at 1:58 PM, Vincent Davis vinc...@vincentdavis.netwrote: In reference to the several comments about [x for x in read] is basically a copy of the entire list. This isn't necessary. or list(read). I had thought I had a problem with having iterators in the takewhile() statement. I thought I testes and it didn't work. It seems I was wrong. It clearly works. I'll make this change and see if it is any better. I actually don't plan to read them all in at once, only as needed, but I do need the whole file in an array to perform some mathematics on them and compare different files. So my interest was in making it faster to open them as needed. I guess part of it is that they are about 5mb so I guess it might be disk speed in part.nks Record your numbers in an array and then work your magic on them later. Don't store the entire file in memory, though. -- Jonathan Gardner jgard...@jonathangardner.net -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to continue after an exception ?
On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki stef.mien...@gmail.com wrote: hello, I would like my program to continue on the next line after an uncaught exception, is that possible ? thanks Stef Mientki That reminds me of VB's On Error Resume Next -- http://mail.python.org/mailman/listinfo/python-list
Re: The Disappearing Program?
W. eWatson wolftra...@invalid.com wrote in news:hlls5c$89...@news.eternal-september.org: I've successfully compiled several small python programs on Win XP into executables using py2exe. A program goes from a name like snowball.py to snowball. A dir in the command prompt window finds snowball.py but not snowball. If I type in snowball, it executes. What's up with that? There is a ludicrous setting in Windows explorer that hides the extensions for known file types, such as .exe. If you see snowball in Win Explorer, that's probably the way your system is set. If you click on Tools . Folder Options . View you should see a checkbox for the Hide extensions for known file types option. Uncheck it, and the disappearing program will return to view. This is a separate issue for why snowball.py executes when you enter snowball at the command line. -- rzed -- http://mail.python.org/mailman/listinfo/python-list
Re: The future of frozen types as the number of CPU cores increases
Chris Rebert c...@rebertia.com wrote: On Thu, Feb 18, 2010 at 11:58 AM, John Nagle na...@animats.com wrote: Python isn't ready for this. Not with the GIL. Is any language, save perhaps Erlang, really ready for it? F# is. I only wish the syntax was a little less Perl-like. Too many special characters. -- Tim Roberts, t...@probo.com Providenza Boekelheide, Inc. -- http://mail.python.org/mailman/listinfo/python-list
Re: The Disappearing Program?
rzed wrote: W. eWatson wolftra...@invalid.com wrote in news:hlls5c$89...@news.eternal-september.org: I've successfully compiled several small python programs on Win XP into executables using py2exe. A program goes from a name like snowball.py to snowball. A dir in the command prompt window finds snowball.py but not snowball. If I type in snowball, it executes. What's up with that? There is a ludicrous setting in Windows explorer that hides the extensions for known file types, such as .exe. If you see snowball in Win Explorer, that's probably the way your system is set. If you click on Tools . Folder Options . View you should see a checkbox for the Hide extensions for known file types option. Uncheck it, and the disappearing program will return to view. The problem with that setting is that an .exe file could masquerade as some other type, eg foo.txt.exe would appear as foo.txt, so you might have thought that you were safe opening it because it's just text. This is a separate issue for why snowball.py executes when you enter snowball at the command line. -- http://mail.python.org/mailman/listinfo/python-list
Efficient way to break up a list into two pieces
Hello, I recently read about augmented assignments and that (with l1, l2 being lists) l1.extend(l2) is more efficient than l1 = l1 + l2 because unnecessary copy operations can be avoided. Now my question is if there's a similar thing for breaking a list into two parts. Let's say I want to remove from l1 everything from and including position 10 and store it in l2. Then I can write l2 = l1[10:] del l1[10:] But since I'm assigning a slice the elements will be copied. Basically, I'm looking for something like l1.pop(10,len(l1)) which returns and removes a whole chunk of data. Is there such a thing (and if not, why not?) Cheers, Martin. -- http://mail.python.org/mailman/listinfo/python-list
Not sure why this is filling my sys memory
Code is below, The files are about 5mb and 230,000 rows. When I have 43 files of them and when I get to the 35th (reading it in) my system gets so slow that it is nearly functionless. I am on a mac and activity monitor shows that python is using 2.99GB of memory (of 4GB). (python 2.6 64bit). The getsizeof() returns 6424 bytes for the alldata . So I am not sure what is happening. Any ideas Thanks *Vincent Davis 720-301-3003 * vinc...@vincentdavis.net my blog http://vincentdavis.net | LinkedInhttp://www.linkedin.com/in/vincentdavis -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On Sat, Feb 20, 2010 at 4:55 PM, marwie mar...@gmx.de wrote: Hello, I recently read about augmented assignments and that (with l1, l2 being lists) l1.extend(l2) is more efficient than l1 = l1 + l2 because unnecessary copy operations can be avoided. Now my question is if there's a similar thing for breaking a list into two parts. Let's say I want to remove from l1 everything from and including position 10 and store it in l2. Then I can write l2 = l1[10:] del l1[10:] But since I'm assigning a slice the elements will be copied. Basically, I'm looking for something like l1.pop(10,len(l1)) which returns and removes a whole chunk of data. Is there such a thing (and if not, why not?) The idiom is: l1, l2 = l1[:10], l1[10:] Don't know if it's optimized or not. If it's not, it could probably be. This is a really common idiom. -- Jonathan Gardner jgard...@jonathangardner.net -- http://mail.python.org/mailman/listinfo/python-list
Re: speed question, reading csv using takewhile() and dropwhile()
On Sat, Feb 20, 2010 at 4:21 PM, Vincent Davis vinc...@vincentdavis.netwrote: Thanks for the help, this is considerably faster and easier to read (see below). I changed it to avoid the break and I think it makes it easy to understand. I am checking the conditions each time slows it but it is worth it to me at this time. It seems you are beginning to understand that programmer time is more valuable than machine time. Congratulations. def read_data_file(filename): reader = csv.reader(open(filename, U),delimiter='\t') data = [] mask = [] outliers = [] modified = [] data_append = data.append mask_append = mask.append outliers_append = outliers.append modified_append = modified.append I know some people do this to speed things up. Really, I don't think it's necessary or wise to do so. maskcount = 0 outliercount = 0 modifiedcount = 0 for row in reader: if '[MASKS]' in row: maskcount += 1 if '[OUTLIERS]' in row: outliercount += 1 if '[MODIFIED]' in row: modifiedcount += 1 if not any((maskcount, outliercount, modifiedcount, not row)): data_append(row) elif not any((outliercount, modifiedcount, not row)): mask_append(row) elif not any((modifiedcount, not row)): outliers_append(row) else: if row: modified_append(row) Just playing with the logic here: 1. Notice that if not row is True, nothing happens? Pull it out explicitly. 2. Notice how it switches from mode to mode? Program it more explicitly. Here's my suggestion: def parse_masks(reader): for row in reader: if not row: continue elif '[OUTLIERS]' in row: parse_outliers(reader) elif '[MODIFIED]' in row: parse_modified(reader) masks.append(row) def parse_outliers(reader): for row in reader: if not row: continue elif '[MODIFIED]' in row: parse_modified(reader) outliers.append(row) def parse_modified(reader): for row in reader: if not row: continue modified.append(row) for row in reader: if not row: continue elif '[MASKS]' in row: parse_masks(reader) elif '[OUTLIERS]' in row: parse_outliers(reader) elif '[MODIFIED]' in row: parse_modified(reader) else: data.append(row) Since there is global state involved, you may want to save yourself some trouble in the future and put the above in a class where separate parsers can be kept separate. It looks like your program is turning into a regular old parser. Any format that is a little more than trivial to parse will need a real parser like the above. -- Jonathan Gardner jgard...@jonathangardner.net -- http://mail.python.org/mailman/listinfo/python-list
Re: speed question, reading csv using takewhile() and dropwhile()
Thanks again for the comment, not sure I will implement all of it but I will separate the if not row The files have some extraneous blank rows in the middle that I need to be sure not to import as blank rows. I am actually having trouble with this filling my sys memory, I posted a separate question Why is this filling my sys memory or something like that is the subject. I might be that my 1yr old son has been trying to help for the last hour. It is very distracting. *Vincent Davis 720-301-3003 * vinc...@vincentdavis.net my blog http://vincentdavis.net | LinkedInhttp://www.linkedin.com/in/vincentdavis On Sat, Feb 20, 2010 at 6:18 PM, Jonathan Gardner jgard...@jonathangardner.net wrote: On Sat, Feb 20, 2010 at 4:21 PM, Vincent Davis vinc...@vincentdavis.netwrote: Thanks for the help, this is considerably faster and easier to read (see below). I changed it to avoid the break and I think it makes it easy to understand. I am checking the conditions each time slows it but it is worth it to me at this time. It seems you are beginning to understand that programmer time is more valuable than machine time. Congratulations. def read_data_file(filename): reader = csv.reader(open(filename, U),delimiter='\t') data = [] mask = [] outliers = [] modified = [] data_append = data.append mask_append = mask.append outliers_append = outliers.append modified_append = modified.append I know some people do this to speed things up. Really, I don't think it's necessary or wise to do so. maskcount = 0 outliercount = 0 modifiedcount = 0 for row in reader: if '[MASKS]' in row: maskcount += 1 if '[OUTLIERS]' in row: outliercount += 1 if '[MODIFIED]' in row: modifiedcount += 1 if not any((maskcount, outliercount, modifiedcount, not row)): data_append(row) elif not any((outliercount, modifiedcount, not row)): mask_append(row) elif not any((modifiedcount, not row)): outliers_append(row) else: if row: modified_append(row) Just playing with the logic here: 1. Notice that if not row is True, nothing happens? Pull it out explicitly. 2. Notice how it switches from mode to mode? Program it more explicitly. Here's my suggestion: def parse_masks(reader): for row in reader: if not row: continue elif '[OUTLIERS]' in row: parse_outliers(reader) elif '[MODIFIED]' in row: parse_modified(reader) masks.append(row) def parse_outliers(reader): for row in reader: if not row: continue elif '[MODIFIED]' in row: parse_modified(reader) outliers.append(row) def parse_modified(reader): for row in reader: if not row: continue modified.append(row) for row in reader: if not row: continue elif '[MASKS]' in row: parse_masks(reader) elif '[OUTLIERS]' in row: parse_outliers(reader) elif '[MODIFIED]' in row: parse_modified(reader) else: data.append(row) Since there is global state involved, you may want to save yourself some trouble in the future and put the above in a class where separate parsers can be kept separate. It looks like your program is turning into a regular old parser. Any format that is a little more than trivial to parse will need a real parser like the above. -- Jonathan Gardner jgard...@jonathangardner.net -- http://mail.python.org/mailman/listinfo/python-list
Re: if not global -- then what?
On Sat, Feb 20, 2010 at 11:25 AM, egasimus fallenbl...@gmail.com wrote: Hi, newbie here. I've read on using the 'global' keyword being discouraged; then what is the preferred way to have something, for example a class containing program settings, accessible from everywhere, in a program spanning multiple files? One of the powerful concepts to come out of Lisp was dynamic scope. This is the ideal solution for configuration and logging. Lexical scope is where the value of the variables are found in the file it was defined in. This is what Python does, and it's what you're used to in most other languages. Dynamic scope is where the value of the variables are found by looking up the call stack. This is similar to perl's local keyword. It's also similar to environment variables, if you consider programs that run programs that run programs as similar to functions. (If you're familiar with neither, then that's fine.) This is useful for a program where you know it needs configuration or logging, but you don't want to have to specify it as part of the program. Whoever calls the program needs to configure it and setup their logger before calling it. Unfortunately, there is no built-in way to deal with dynamic variables in Python. But that doesn't mean you can't emulate it explicitly and clearly! In order to emulate it, you simply pass the configuration and such down to the functions you call. Each function or class instance needs to pass it further along. So, rather than: def foo(): bar() def bar() ... you need to have: def foo(config): bar(config) def bar(config): ... Obviously, only certain functions care for the config and logging setup. Others don't. And obviously, you can modify the value on the way down. Just be sure to make a copy or you'll change the entire stack's value for that configuration. There are a couple of systems that do something close to this. I don't know of any, except SQLAlchemy, that does this exactly right. WSGI is also a system that does this right, although I don't know that many people realize it. Usually, what people end up doing is setting the config values to some global in some module. Then everyone is expected to look in that module for the config. This ends up being messy, particularly for testing where you'd like to mess with the config without messing with THE config. You can tell it's messy because people end up writing code like this: old_value = get_config(...) set_config(... some new value ...) ... do your stuff ... set_config(...back to old_value...) This kind of code is hard to get right. (What happens if there is an exception before you set the old config back? That's right, you need a try-finally block.) In terms of global, you should only really use global when you are need to assign to a lexically scoped variable that is shared among other functions. For instance: def foo(): i = 0 def inc(): global i; i+=1 def dec(): global i; i-=1 def get(): return i return (inc, dec, get) This really isn't that common, although it is useful. Note that the above might be better organized into a class instance. Good luck. -- Jonathan Gardner jgard...@jonathangardner.net -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On Sat, 20 Feb 2010 16:55:10 -0800, marwie wrote: Hello, I recently read about augmented assignments and that (with l1, l2 being lists) l1.extend(l2) is more efficient than l1 = l1 + l2 because unnecessary copy operations can be avoided. Now my question is if there's a similar thing for breaking a list into two parts. Let's say I want to remove from l1 everything from and including position 10 and store it in l2. Then I can write l2 = l1[10:] del l1[10:] But since I'm assigning a slice the elements will be copied. Yes, list slicing can't make any sort of assumptions about what you're going to do next, so when you take a slice, it has to copy the data. Basically, I'm looking for something like l1.pop(10,len(l1)) which returns and removes a whole chunk of data. Is there such a thing (and if not, why not?) Such a list pop would have to copy the data too. How else could it work? What exactly is the problem you are trying to solve? If you are unhappy that copy-and-remove a slice from a list takes two lines instead of one (won't somebody think of the shortage of newlines!!! *wink*), then just write a helper function and call that: def copy_and_remove(alist, slice): tmp = alist[slice] del alist[slice] return tmp l2 = copy_and_remove(l1, slice(10, None)) If you are unhappy that copying slices from lists copies data, well that's just the way things are. Don't make the mistake of trying to optimize your application before you actually know what bits need optimizing. Python lists are arrays of pointers to objects, so copying a slice is fast: it doesn't have to copy the objects, just pointers. Deleting from the end of the list is also quick, because you don't have to move memory, just clear some pointers and change the length field. Splitting such an array without copying data is, essentially, impossible. Python lists aren't linked lists. But as I said, most of the time copying slices doesn't matter: the time taken is unlikely to be a serious factor in practice. If it is (and you've profiled your code, right? you're not just guessing what you think is fast and slow?) then there may be ways to optimize your code to avoid needing to copy slices, but we'd need to know what you're doing with the data. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: /usr/bin/ld: cannot find -lz on Cent OS - Python 2.4
On Sat, Feb 20, 2010 at 3:03 PM, V8 NUT olaye1...@googlemail.com wrote: /usr/bin/ld: skipping incompatible /usr/lib/libz.so when searching for -lz /usr/bin/ld: skipping incompatible /usr/lib/libz.a when searching for - lz /usr/bin/ld: cannot find -lz This is your problem. Am trying to get MySQL-python-1.2.3c1 installed on a CentOS 5 box running Python 2.4 I do not want to upgrade my Python version. I have tried: 1. yum install mysql-python 2. yum install librsync-devel 3. yum install python-devel With all return no errors and a notification that there's nothing to install or update. I think you need libz's -devel module. Not sure what libz is or what its name is in the yum repository. Might take a search or maybe an rpm -qif /usr/lib/libz.so The problem could be that mysql-python expects to see a different version of libz than what's installed on your box. If so, you'll need to see if you can find the right version. The MySQL-python people should have more help for you. -- Jonathan Gardner jgard...@jonathangardner.net -- http://mail.python.org/mailman/listinfo/python-list
Re: datelib pythonification
hello all,
since I posted this last time, I've added a new function dates_diff and
modified the dates_dict function to set timedelta values returned by
dates_diff in the returned dict
def dates_dict(self,*targs,**dargs):
dates_dict() - takes params same as prefs()
returns dict key/value pair of formatted/calculated dates
key is a date, value is timedelta enddate-startdate
self.prefs(*targs,**dargs)
d={}
try:
if self.format!=:
d[self.startdate.strftime(self.format)]=self.dates_diff()
else:
d[self.startdate]=self.dates_diff()
except ValueError:
d[%s%self.startdate]=self.dates_diff()
while self.startdateself.enddate:
a=self.calc(*targs,**dargs)[0]
d[a]=self.dates_diff()
self.reset_dates()
return d
def dates_diff(self,*targs):
dates_diff - return timedelta difference between startdata and
enddate
takes nothing, a tuple, a list or 2 date objects as params
return enddate - startdate timedelta
start,end=(self.startdate,self.enddate)
if targs:
if(len(targs)==1):
if(type(targs[0])==type(()) or targs[0] == type([])):
if(len(targs[0])==2):
start,end=(targs[0][0],targs[0][1])
elif(len(targs)== 2):
if(type(targs[0])==type(date.today()) and targs[1] ==
type(date.today())):
start,end=(targs[0],targs[1])
return end-start
-Alex Goretoy
--
http://mail.python.org/mailman/listinfo/python-list
Re: Not sure why this is filling my sys memory
On Sat, Feb 20, 2010 at 5:07 PM, Vincent Davis vinc...@vincentdavis.net wrote: Code is below, The files are about 5mb and 230,000 rows. When I have 43 files of them and when I get to the 35th (reading it in) my system gets so slow that it is nearly functionless. I am on a mac and activity monitor shows that python is using 2.99GB of memory (of 4GB). (python 2.6 64bit). The getsizeof() returns 6424 bytes for the alldata . So I am not sure what is happening. With this kind of data set, you should start looking at BDBs or PostgreSQL to hold your data. While processing files this large is possible, it isn't easy. Your time is better spent letting the DB figure out how to arrange your data for you. -- Jonathan Gardner jgard...@jonathangardner.net -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On Sat, 20 Feb 2010 17:06:36 -0800, Jonathan Gardner wrote: On Sat, Feb 20, 2010 at 4:55 PM, marwie mar...@gmx.de wrote: [...] l2 = l1[10:] del l1[10:] But since I'm assigning a slice the elements will be copied. Basically, I'm looking for something like l1.pop(10,len(l1)) which returns and removes a whole chunk of data. Is there such a thing (and if not, why not?) The idiom is: l1, l2 = l1[:10], l1[10:] No, that has different behaviour to what the Original Poster wants, AND it does two copies instead of one: (1) copy l1[:10] and l1[10:] (2) assign the names l1 and l2 to them (3) if, and only if, there are no other references to the original l1, it gets deleted (garbage collected). What the OP is doing is quite different: (1) copy l1[:10] (2) assign the name l2 to it (3) resize l1 in place to the first 10 items. What the OP wants is: (1) assign the name l2 to l1[:10] without copying (2) resize l1 in place to the first 10 items without affecting l2. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
netcdf4-python
I'm trying to use the python module netcdf4-python to read a netcdf
file. So far I'm trying to do the basics and just open the script:
from netCDF4 import Dataset
rootgrp = Dataset('20060402-201025.netcdf', 'r',
format='NETCDF3_CLASSIC')
print rootgrp.file_format
rootgrp.close()
when I do this I get the exit code error (when I run in Scite):
pythonw -u netcdf_test.py
Exit code: -1073741819
or Python stops responding (in windows cmd).
I'm not sure what is wrong so if anyone as any ideas I would gladly
send you the netcdf file to try. Thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Re: if not global -- then what?
On Sat, 20 Feb 2010 17:34:15 -0800, Jonathan Gardner wrote: In terms of global, you should only really use global when you are need to assign to a lexically scoped variable that is shared among other functions. For instance: def foo(): i = 0 def inc(): global i; i+=1 def dec(): global i; i-=1 def get(): return i return (inc, dec, get) That doesn't do what you think it does: def foo(): ... i = 0 ... def inc(): global i; i+=1 ... def dec(): global i; i-=1 ... def get(): return i ... return (inc, dec, get) ... inc = foo()[0] inc() Traceback (most recent call last): File stdin, line 1, in module File stdin, line 3, in inc NameError: global name 'i' is not defined The problem is that i is not global. Inside the inc and dec functions, you need to declare i nonlocal, not global, and that only works in Python 3 or better. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On 21 Feb., 02:30, Steven D'Aprano st...@remove-this- cybersource.com.au wrote: Python lists are arrays of pointers to objects, so copying a slice is fast: it doesn't have to copy the objects, just pointers. Deleting from the end of the list is also quick, because you don't have to move memory, just clear some pointers and change the length field. Splitting such an array without copying data is, essentially, impossible. Python lists aren't linked lists. Well, to split a C array I would simply set l2 to point to l1[10] and then change the length of l1 (which I store somewhere else). No copying of elements needed. I would have assumed that python can do something like this with its internal arrays of pointers, too. Anyway, this was more a question about coding style. I use l1.extend(l2) or l1 += l2 rather than l1 = l1 + l2 because it's as readable and possibly faster. I was simply wondering if something similar exists for splitting lists. -- http://mail.python.org/mailman/listinfo/python-list
Re: The future of frozen types as the number of CPU cores increases
On Feb 18, 2:58 pm, John Nagle na...@animats.com wrote: Multiple processes are not the answer. That means loading multiple copies of the same code into different areas of memory. The cache miss rate goes up accordingly. A decent OS will use copy-on-write with forked processes, which should carry through to the cache for the code. -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to continue after an exception ?
On 02/21/10 12:02, Stef Mientki wrote:
On 21-02-2010 01:21, Lie Ryan wrote:
On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki
stef.mien...@gmail.com wrote:
hello,
I would like my program to continue on the next line after an uncaught
exception,
is that possible ?
thanks
Stef Mientki
That reminds me of VB's On Error Resume Next
I think that's what I'm after ...
First, read this:
http://www.developerfusion.com/code/4325/on-error-resume-next-considered-harmful/
I already redirected sys.excepthook to my own function,
but now I need a way to get to continue the code on the next line.
Is that possible ?
No, not in python. You can (ab)use generators' yield to resume
execution, but not in the general case:
def on_error_resume_next(func):
def _func(*args, **kwargs):
gen = func(*args, **kwargs)
resp = next(gen)
while isinstance(resp, Exception):
print 'an error happened, ignoring...'
resp = next(gen)
return resp
return _func
@on_error_resume_next
def add_ten_error_if_zero(args):
if args == 0:
# raise Exception()
yield Exception()
# return args + 10
yield args + 10
print add_ten_error_if_zero(0)
print add_ten_error_if_zero(10)
A slightly better approach is to retry calling the function again, but
as you can see, it's not appropriate for certain cases:
def retry_on_error(func):
def _func(*args, **kwargs):
while True:
try:
ret = func(*args, **kwargs)
except Exception:
print 'An error happened, retrying...'
else:
return ret
return _func
@retry_on_error
def add_ten_error_if_zero(args):
if args == 0:
raise Exception()
return args + 10
print add_ten_error_if_zero(0)
print add_ten_error_if_zero(10)
A much better approach is to use callbacks, the callbacks determines
whether to raise an exception or continue execution:
def handler(e):
if datetime.datetime.now() = datetime.datetime(2012, 12, 21):
raise Exception('The world has ended')
# else: ignore, it's fine
def add_ten_error_if_zero(args, handler):
if args == 0:
handler(args)
return args + 10
print add_ten_error_if_zero(0, handler)
print add_ten_error_if_zero(10, handler)
print add_ten_error_if_zero(0, lambda e: None) # always succeeds
Ignoring arbitrary error is against the The Zen of Python Errors should
never pass silently.; not that it is ever a good idea to ignore
arbitrary error, when an exception happens often the function is in an
indeterminate state, and continuing blindly could easily cause havocs.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to continue after an exception ?
On Feb 20, 2010, at 9:17 PM, Lie Ryan wrote: On 02/21/10 12:02, Stef Mientki wrote: On 21-02-2010 01:21, Lie Ryan wrote: On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki stef.mien...@gmail.com wrote: hello, I would like my program to continue on the next line after an uncaught exception, is that possible ? thanks Stef Mientki That reminds me of VB's On Error Resume Next I think that's what I'm after ... First, read this: http://www.developerfusion.com/code/4325/on-error-resume-next-considered-harmful/ The link goes to an Oh dear. Gremlins at work! page. They're probably using On Error Resume Next in their VBScript code and this is the last resort page ;-). S -- http://mail.python.org/mailman/listinfo/python-list
Re: The future of frozen types as the number of CPU cores increases
sjdevn...@yahoo.com wrote: On Feb 18, 2:58 pm, John Nagle na...@animats.com wrote: Multiple processes are not the answer. That means loading multiple copies of the same code into different areas of memory. The cache miss rate goes up accordingly. A decent OS will use copy-on-write with forked processes, which should carry through to the cache for the code. That doesn't help much if you're using the subprocess module. The C code of the interpreter is shared, but all the code generated from Python is not. This will get even worse when Unladen Swallow starts generating vast swaths of unshared x86 instructions. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to continue after an exception ?
On Sun, 2010-02-21 at 13:17 +1100, Lie Ryan wrote:
On 02/21/10 12:02, Stef Mientki wrote:
On 21-02-2010 01:21, Lie Ryan wrote:
On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki
stef.mien...@gmail.com wrote:
hello,
I would like my program to continue on the next line after an uncaught
exception,
is that possible ?
thanks
Stef Mientki
That reminds me of VB's On Error Resume Next
I think that's what I'm after ...
A much better approach is to use callbacks, the callbacks determines
whether to raise an exception or continue execution:
def handler(e):
if datetime.datetime.now() = datetime.datetime(2012, 12, 21):
raise Exception('The world has ended')
# else: ignore, it's fine
def add_ten_error_if_zero(args, handler):
if args == 0:
handler(args)
return args + 10
print add_ten_error_if_zero(0, handler)
print add_ten_error_if_zero(10, handler)
print add_ten_error_if_zero(0, lambda e: None) # always succeeds
Or if you don't like having to explicitly manage callbacks, you can try
the withrestart module:
http://pypi.python.org/pypi/withrestart/
It tries to pinch some of the good ideas from Common Lisp's
error-handling system.
from withrestart import *
def add_ten_error_if_zero(n):
# This gives calling code the option to ignore
# the error, or raise a different one.
with restarts(skip,raise_error):
if n == 0:
raise ValueError
return n + 10
# This will raise ValueError
print add_ten_error_if_zero(0)
# This will print 10
with Handler(ValueError,skip):
print add_ten_error_if_zero(0)
# This will exit the python interpreter
with Handler(ValueError,raise_error,SystemExit):
print add_ten_error_if_zero(0)
Cheers,
Ryan
--
Ryan Kelly
http://www.rfk.id.au | This message is digitally signed. Please visit
r...@rfk.id.au| http://www.rfk.id.au/ramblings/gpg/ for details
signature.asc
Description: This is a digitally signed message part
--
http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On 21 Feb., 02:41, Steven D'Aprano st...@remove-this- cybersource.com.au wrote: What the OP is doing is quite different: (1) copy l1[:10] (2) assign the name l2 to it (3) resize l1 in place to the first 10 items. What the OP wants is: (1) assign the name l2 to l1[:10] without copying (2) resize l1 in place to the first 10 items without affecting l2. Assuming you meant l1[10:] instead of l1[:10], that's what I wanted, yes. -- http://mail.python.org/mailman/listinfo/python-list
Re: Not sure why this is filling my sys memory
Here is a sample of the output, It almost instantly uses 2GB and then starts using VMem. This is probably the right suggestion but it's another thing to install It's probably also worth being aware of guppy's heapy stuff: http://guppy-pe.sourceforge.net/heapy_tutorialhttp://guppy-pe.sourceforge.net/heapy_tutorial.htmlI find it quite nice to have the following to get a quick point-in-time estimate of my app's memory usage: print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024)) 19 files of 43 using 3352 bytes of memory Loading into memory: 3_veg_nm_rep1.txt 3_veg_nm_rep1.txt has 228484 rows of data 3_veg_nm_rep1.txt has 0 rows of masked data 3_veg_nm_rep1.txt has 141 rows of outliers 3_veg_nm_rep1.txt has 0 modified rows of data 280bytes of memory used for 3_veg_nm_rep1.txt 20 files of 43 using 3352 bytes of memory Loading into memory: 3_veg_nm_rep2.txt 3_veg_nm_rep2.txt has 228484 rows of data 3_veg_nm_rep2.txt has 0 rows of masked data 3_veg_nm_rep2.txt has 119 rows of outliers 3_veg_nm_rep2.txt has 0 modified rows of data 280bytes of memory used for 3_veg_nm_rep2.txt 21 files of 43 using 3352 bytes of memory Loading into memory: 3_veg_phd_rep1.txt 3_veg_phd_rep1.txt has 228484 rows of data 3_veg_phd_rep1.txt has 0 rows of masked data 3_veg_phd_rep1.txt has 63 rows of outliers 3_veg_phd_rep1.txt has 0 modified rows of data 280bytes of memory used for 3_veg_phd_rep1.txt 22 files of 43 using 6424 bytes of memory Loading into memory: 3g_c285-11.txt 3g_c285-11.txt has 228484 rows of data 3g_c285-11.txt has 0 rows of masked data 3g_c285-11.txt has 65 rows of outliers 3g_c285-11.txt has 0 modified rows of data 280bytes of memory used for 3g_c285-11.txt 23 files of 43 using 6424 bytes of memory Loading into memory: 3g_c285-42.txt 3g_c285-42.txt has 228484 rows of data 3g_c285-42.txt has 0 rows of masked data 3g_c285-42.txt has 27 rows of outliers 3g_c285-42.txt has 0 modified rows of data 280bytes of memory used for 3g_c285-42.txt 24 files of 43 using 6424 bytes of memory Loading into memory: A6AF.txt A6AF.txt has 228484 rows of data A6AF.txt has 0 rows of masked data A6AF.txt has 36 rows of outliers A6AF.txt has 0 modified rows of data 280bytes of memory used for A6AF.txt 25 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3026_rep1.txt Grigg_3026_rep1.txt has 228484 rows of data Grigg_3026_rep1.txt has 0 rows of masked data Grigg_3026_rep1.txt has 949 rows of outliers Grigg_3026_rep1.txt has 0 modified rows of data 280bytes of memory used for Grigg_3026_rep1.txt 26 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3026_rep2.txt Grigg_3026_rep2.txt has 228484 rows of data Grigg_3026_rep2.txt has 0 rows of masked data Grigg_3026_rep2.txt has 361 rows of outliers Grigg_3026_rep2.txt has 0 modified rows of data 280bytes of memory used for Grigg_3026_rep2.txt 27 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3026_rep3_both.txt Grigg_3026_rep3_both.txt has 228484 rows of data Grigg_3026_rep3_both.txt has 0 rows of masked data Grigg_3026_rep3_both.txt has 41 rows of outliers Grigg_3026_rep3_both.txt has 0 modified rows of data 280bytes of memory used for Grigg_3026_rep3_both.txt 28 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3131_rep1.txt Grigg_3131_rep1.txt has 228484 rows of data Grigg_3131_rep1.txt has 0 rows of masked data Grigg_3131_rep1.txt has 537 rows of outliers Grigg_3131_rep1.txt has 0 modified rows of data 280bytes of memory used for Grigg_3131_rep1.txt 29 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3131_rep2.txt Grigg_3131_rep2.txt has 228484 rows of data Grigg_3131_rep2.txt has 0 rows of masked data Grigg_3131_rep2.txt has 238 rows of outliers Grigg_3131_rep2.txt has 0 modified rows of data 280bytes of memory used for Grigg_3131_rep2.txt 30 files of 43 using 6424 bytes of memory *Vincent Davis 720-301-3003 * vinc...@vincentdavis.net my blog http://vincentdavis.net | LinkedInhttp://www.linkedin.com/in/vincentdavis On Sat, Feb 20, 2010 at 7:40 PM, sstein...@gmail.com sstein...@gmail.comwrote: On Feb 20, 2010, at 9:21 PM, Vincent Davis wrote: See this article for some more info about the reported sizes of things: http://www.doughellmann.com/PyMOTW/sys/limits.html http://www.doughellmann.com/PyMOTW/sys/limits.htmlNice article but I must have missed something useful to my current issue. Do I get any hints? Oh, sorry, there was the part about getsizeof() not including attributes unless the class supplies a __sizeof__ method and a comment at the bottom: It's probably also worth being aware of guppy's heapy stuff: http://guppy-pe.sourceforge.net/heapy_tutorialhttp://guppy-pe.sourceforge.net/heapy_tutorial.html I find it quite nice to have the following to get a quick point-in-time estimate of my app's memory usage: print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024)) Put a few of those in at various places in the collection
Re: netcdf4-python
On Feb 20, 7:47 pm, deadpickle deadpic...@gmail.com wrote:
I'm trying to use the python module netcdf4-python to read a netcdf
file. So far I'm trying to do the basics and just open the script:
from netCDF4 import Dataset
rootgrp = Dataset('20060402-201025.netcdf', 'r',
format='NETCDF3_CLASSIC')
print rootgrp.file_format
rootgrp.close()
when I do this I get the exit code error (when I run in Scite):
pythonw -u netcdf_test.py
Exit code: -1073741819
or Python stops responding (in windows cmd).
I'm not sure what is wrong so if anyone as any ideas I would gladly
send you the netcdf file to try. Thanks.
If the file is really of NetCDF3 format, scipy.io.netcdf should work.
Replace
netCDF4.Dataset(filename,'r',format='NETCDF3_CLASSIC')
with
scipy.io.netcdf.netcdf_open(filename,'r')
--Matt Newville
--
http://mail.python.org/mailman/listinfo/python-list
lists of variables
I'm relatively new to python and I was very surprised by the following behavior:
a=1
b=2
mylist=[a,b]
print mylist
[1, 2]
a=3
print mylist
[1, 2]
Whoah! Are python lists only for literals? Nope:
c={}
d={}
mydlist=[c,d]
print mydlist
[{}, {}]
c['x']=1
print mydlist
[{'x': 1}, {}]
So it looks like variables in a list are stored as object references.
This seems to confirm that:
mydlist[1]['y']=4
print mydlist
[{}, {'y': 4}]
So I figure my initial example doesn't work because if you assign a
literal to something it is changing the object. But modifying a list
or dict (as long as you don't re-construct it) does not change the
object.
I can think of some ways to work around this, including using single
element lists as pointers:
aa=[1]
bb=[2]
myplist=[aa,bb]
print myplist
[[1], [2]]
aa[0]=3
print myplist
[[3], [2]]
But what would be the python way to accomplish list of variables
functionality?
--
http://mail.python.org/mailman/listinfo/python-list
Re: lists of variables
On Sat, Feb 20, 2010 at 7:25 PM, Michael Pardee
python-l...@open-sense.com wrote:
I'm relatively new to python and I was very surprised by the following
behavior:
a=1
b=2
mylist=[a,b]
print mylist
[1, 2]
a=3
print mylist
[1, 2]
Whoah! Are python lists only for literals? Nope:
c={}
d={}
mydlist=[c,d]
print mydlist
[{}, {}]
c['x']=1
print mydlist
[{'x': 1}, {}]
So it looks like variables in a list are stored as object references.
This seems to confirm that:
mydlist[1]['y']=4
print mydlist
[{}, {'y': 4}]
So I figure my initial example doesn't work because if you assign a
literal to something it is changing the object. But modifying a list
or dict (as long as you don't re-construct it) does not change the
object.
Correct. If you want more gory details, read
http://effbot.org/zone/call-by-object.htm
I can think of some ways to work around this, including using single
element lists as pointers:
aa=[1]
bb=[2]
myplist=[aa,bb]
print myplist
[[1], [2]]
aa[0]=3
print myplist
[[3], [2]]
But what would be the python way to accomplish list of variables
functionality?
What do you need that functionality for exactly? It's a rather low-level notion.
Cheers,
Chris
--
http://blog.rebertia.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On Sat, 20 Feb 2010 17:55:18 -0800, marwie wrote: On 21 Feb., 02:30, Steven D'Aprano st...@remove-this- cybersource.com.au wrote: Python lists are arrays of pointers to objects, so copying a slice is fast: it doesn't have to copy the objects, just pointers. Deleting from the end of the list is also quick, because you don't have to move memory, just clear some pointers and change the length field. Splitting such an array without copying data is, essentially, impossible. Python lists aren't linked lists. Well, to split a C array I would simply set l2 to point to l1[10] and then change the length of l1 (which I store somewhere else). No copying of elements needed. I would have assumed that python can do something like this with its internal arrays of pointers, too. Python lists aren't C arrays either. Python lists are *objects*. Everything in Python is an object. Consequently, Python lists have a header, which includes the len. You don't store the length somewhere else, you store it in the object: this makes for less complexity. You can't just point l2 at an arbitrary index in an array and expect it to work, it needs a header. Additionally, Python lists are over-allocated so that appends are fast. A list of (say) 1000 items might be over-allocated to (say) 1024 items, so that you can do 24 appends before the array is full and the array needs to be resized. This means that, on average, Python list appending is O(1) instead of O(N). You can't just change the length blindly, you need to worry about the over-allocation. I'm sympathetic to your concern: I've often felt offended that doing something like this: x = SomeReallyBigListOrString for item in x[1:]: process(item) has to copy the entire list or string (less the first item). But honestly, I've never found a situation where it actually mattered. Anyway, this was more a question about coding style. I use l1.extend(l2) or l1 += l2 rather than l1 = l1 + l2 because it's as readable and possibly faster. I really, really hate augmented assignment for everything other than ints and floats because you can't predict what it will do. Take this for example: mylist = [1, 2, 3, 4] same_again = mylist mylist.append(5) same_again [1, 2, 3, 4, 5] mylist and same_again are the same object, so append and extend behave predictably and simply. So is simple too: mylist = mylist + [-1, -2, -3] mylist [1, 2, 3, 4, 5, -1, -2, -3] same_again [1, 2, 3, 4, 5] Because you have re-bound the name mylist to a new list, same_again doesn't get modified. Nice. But what about mylist += something else? Is it an in-place modification, like extend and append, or a rebinding, like +? Who can remember? The answer will depend on whether mylist is a builtin list, or a subclass. And then there's this mystery: mylist = [1, 2, 3] mytuple = (mylist, None) mytuple[0] += [4] Traceback (most recent call last): File stdin, line 1, in module TypeError: 'tuple' object does not support item assignment mylist [1, 2, 3, 4] So in fact, += is *always* a rebinding, but *sometimes* it is an in-place modification as well. Yuck. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: lists of variables
On Sat, Feb 20, 2010 at 7:25 PM, Michael Pardee python-l...@open-sense.comwrote: But what would be the python way to accomplish list of variables functionality? The problem is... Python doesn't have variables. At least not in the way that you may be used to from other languages. Yeah, it's got data, and data obviously has to be identifiable in some way, but they aren't put together like variables are. A variable is a box with a name on it, and in that context a 'list of variables' makes sense. You're moving around the box, or referencing the box itself. It may happen to have something in it, but that 'something' doesn't really have an identity of its own. Its just.. the thing that's in box-A. In Python, everything's an object. And objects don't really have identities of their own(okay, they do, but its not meaningful except to an identity test), but they can be named. They object has no flabbering clue what its named, or how many different people know its name, but hey. a = 1 In many languages, that's making a variable a and assigning it the value of 1. In Python, that's making an int object with a value of 1, and giving it a name in the current namespace. This -isn't- just using dumber/simpler words to say the same thing :) All(*) python namespaces are really dictionaries behind the scenes. a = 1 is actually doing, current_namespaces[a] = 1. Those may sound similar, but they're really not: most importantly, because the a in another language is a -thing- unto itself which may hold an object. But a in Python is just a label in some dictionary somewhere. You can get the object a is naming, but you can't get /a/ itself. Its not a 'thing'. There's no way to make a list of variables, because /variables/ are the part that's really missing. If you do: lst = [a, b] Python doesn't create a list and put two variables in it. It creates a list, and while doing so, sees that we want to pass 'a' and 'b' into that list. So it grabs the actual objects a and b are naming, and puts them in the list. So, long story short: what's the python way to accomplish a list of variables? Well... IMHO, it's to not do so. : ) Make a dict, pass the dict around, there's your list of variables. True, you can't ever access them /as/ variables, and have to access them as dict items, but... that's the Python way to do that. If order matters, do an ordered dict. HTH, --S *: Yes, I know about local namespaces being optimized in CPython to be an array-like structure. Implementation detail. -- http://mail.python.org/mailman/listinfo/python-list
Re: The future of frozen types as the number of CPU cores increases
John Nagle na...@animats.com writes: A decent OS will use copy-on-write with forked processes, which should carry through to the cache for the code. That doesn't help much if you're using the subprocess module. The C code of the interpreter is shared, but all the code generated from Python is not. Emacs in days of yore used a hack called unexec to bring its Lisp code into the pure text segment. I think it's not used any more because machines are now big enough that the benefits aren't that great. Basically in the Emacs build process, you'd start up the C portion of Emacs, then tell it to load all its Lisp code, then call unexec. Unexec basically performed a core dump of the process, then ran something that manipulated the original Emacs image and the core dump into a new executable that had the static Lisp code and data now marked as part of the (immutable) program area. Unexec necessarily had platform dependencies, but this was managed with a bunch of ifdefs in a reasonably clean and maintainable way. I could imagine doing something similar with a large Python program. -- http://mail.python.org/mailman/listinfo/python-list
Re: lists of variables
Michael Pardee python-l...@open-sense.com writes: But what would be the python way to accomplish list of variables functionality? You'll need to explain what “list of variables” functionality is. If you mean “collection of name-to-value mappings”, the native mapping type in Python is ‘dict’. If that doesn't meet your needs, you'll need to be more specific about what behaviour you want. -- \“Most people don't realize that large pieces of coral, which | `\ have been painted brown and attached to the skull by common | _o__)wood screws, can make a child look like a deer.” —Jack Handey | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On Sat, Feb 20, 2010 at 6:55 PM, marwie mar...@gmx.de wrote: Now my question is if there's a similar thing for breaking a list into two parts. Let's say I want to remove from l1 everything from and including position 10 and store it in l2. Then I can write l2 = l1[10:] del l1[10:] With Python's list implementation, there's no faster way to split a list into two parts. Because Python lists are essentially variable-length arrays, Python has to copy O(n) references to create l2 and then decrement O(n) references to remove them from l1. I wrote an extension type called the blist to handle this and other problems. You use it just like you'd use a list, but it has different performance characteristics. It uses a hybrid array-tree structure and performs copy-on-write behind the scenes, so it takes only O(log n) time to create l2 and only O(log n) time to trim l1. You'd use it something like this: from blist import blist l1 = blist() # Here, populate l1 as you normally would l2 = l1[10:] del l1[10:] It's available at: http://pypi.python.org/pypi/blist/ -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com -- http://mail.python.org/mailman/listinfo/python-list
Re: lists of variables
On Sat, 20 Feb 2010 21:25:19 -0600, Michael Pardee wrote: I'm relatively new to python and I was very surprised by the following behavior: [snip] I don't see why. It's fairly unusual behaviour to want, and it would be surprising if you did this: def test(): x = 1 mylist = [2, 4, x] function(mylist) assert x == 1 and the assertion failed, even though you never passed x to the function. Such behaviour could easily turn into a never-ending source of bugs. So it looks like variables in a list are stored as object references. Python doesn't store variables in lists, it stores objects, always. Even Python variables aren't variables *grin*, although it's really difficult to avoid using the term. Python variables are mappings between names (strings) and objects, not memory locations. So I figure my initial example doesn't work because if you assign a literal to something it is changing the object. But modifying a list or dict (as long as you don't re-construct it) does not change the object. Yes, we talk about name binding (and rebinding) versus mutation. A simple example: alist = blist = [] # bind two names to the same list object alist.append(1) # mutate the list (modify in place) blist [1] alist = alist + [2] # a rebinding operation blist [1] alist [1, 2] I can think of some ways to work around this, including using single element lists as pointers: Yes, that's a standard way to do it, except that by standard I mean really really really rare, honestly, hardly anyone does that. Slightly less rare, but still uncommon, is to wrap objects in an instance: class Record: pass o = Record() o.x = 1 o.y = 2 modify(o) print o.x, o.y Of course you can make the class as fancy, or as simple, as you want. But a better approach is to take advantage of Python's ability to return multiple values: x = 1 y = 2 x, y = modify(x, y) rather than: x = 1 y = 2 modify([x, y]) Speaking as an old Pascal coder, you won't miss pass-by-reference very often. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
Steven D'Aprano wrote: [snip] I'm sympathetic to your concern: I've often felt offended that doing something like this: x = SomeReallyBigListOrString for item in x[1:]: process(item) has to copy the entire list or string (less the first item). But honestly, I've never found a situation where it actually mattered. [snip] Could lists gain an .items() method with start and end positions? I was thinking that it would be a 'view', like with dicts in Python 3. Of course, that would only be worthwhile with very long lists: x = SomeReallyBigListOrString for item in x.items(1): process(item) -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On Sat, Feb 20, 2010 at 5:41 PM, Steven D'Aprano st...@remove-this-cybersource.com.au wrote: What the OP wants is: (1) assign the name l2 to l1[:10] without copying (2) resize l1 in place to the first 10 items without affecting l2. For ten items, though, is it really faster to muck around with array lengths than just copying the data over? Array copies are extremely fast on modern processors. Programmer time and processor time being what it is, I still think my answer is the correct one. No, it's not what he says he wants, but it is what he needs. -- Jonathan Gardner jgard...@jonathangardner.net -- http://mail.python.org/mailman/listinfo/python-list
Re: lists of variables
On Sat, Feb 20, 2010 at 7:25 PM, Michael Pardee
python-l...@open-sense.com wrote:
But what would be the python way to accomplish list of variables
functionality?
You're looking for namespaces, AKA dicts.
vars = {}
vars['a'] = 1
vars['b'] = 2
mylist = ['a', 'b']
print [vars[i] for i in mylist] # Here's your dereference
[1,2]
vars['a'] = 3
print [vars[i] for i in mylist]
[3,2]
--
Jonathan Gardner
jgard...@jonathangardner.net
--
http://mail.python.org/mailman/listinfo/python-list
