Re: multiprocessing speedup
Thanks for the responses. I will create another thread to supply a more realistic example. On Tue, Sep 29, 2015 at 10:12 AM, Oscar Benjamin <oscar.j.benja...@gmail.com > wrote: > On Tue, 29 Sep 2015 at 02:22 Rita <rmorgan...@gmail.com> wrote: > >> I am using the multiprocessing with apply_async to do some work. Each >> task takes a few seconds but I have several thousand tasks. I was wondering >> if there is a more efficient method and especially when I plan to operate >> on a large memory arrays (numpy) >> > >> Here is what I have now >> > import multiprocessing as mp >> import random >> >> def f(x): >> count=0 >> for i in range(x): >> x=random.random() >> y=random.random() >> if x*x + y*y<=1: >> count+=1 >> >> return count >> > > I assume you're using the code shown as a toy example of playing with the > multiprocessing module? If not then the function f can be made much more > efficient. > > The problem is that while it's good that you have distilled your problem > into a simple program for testing it's not really possible to find a more > efficient way without finding the bottleneck which means looking at the > full problem. > > -- > Oscar > -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
multiprocessing speedup
I am using the multiprocessing with apply_async to do some work. Each task takes a few seconds but I have several thousand tasks. I was wondering if there is a more efficient method and especially when I plan to operate on a large memory arrays (numpy) Here is what I have now import multiprocessing as mp import random def f(x): count=0 for i in range(x): x=random.random() y=random.random() if x*x + y*y<=1: count+=1 return count def main(): resultObj=[] n=1 P=mp.Pool(2) for arg in xrange(n): resultObj.append(P.apply_async(f,(arg,))) P.close() P.join() result = [ i.get() for i in resultObj ] print sum(result)/(n) if __name__=="__main__": main() 1) Does multiprocessing do a fork for each task? 2) If so, I assume thats costly due to setup and teardown. Would this be the case? 3) I plan to pass large arrays to function,f, therefore is there a more efficient method to achieve this? -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On Thu, Jul 3, 2014 at 2:54 AM, Mark Lawrence breamore...@yahoo.co.uk wrote: On 03/07/2014 02:17, Rita wrote: On Wed, Jul 2, 2014 at 2:46 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl wrote: On 2-7-2014 4:04, Rita wrote: yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. (Please don't top-post, it makes the discussion harder to follow.) On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list [...] Why would the fact that you upgrade or remove a package, affect another developer in your group? Are you all using the same machine to develop on, with one Python installation? I think you'll have to tell us some more details about the way you work together before we can give a meaningful answer to your question. Irmen -- https://mail.python.org/mailman/listinfo/python-list we have a shared mount point which has our python install. we have 3 servers on one part of the campus and 2 in another part. I want to find out what packages our user base is using thats the final goal. I can figure out who is using python by writing a wrapper but not what module. -- --- Get your facts first, then you can distort them as you please.-- You can check every users's program for import statements but do you really need to, why not check what's in the site-packages folder for your python install? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list how can i get frequency of the module usage? thats the end goal. -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On Thu, Jul 3, 2014 at 8:36 AM, Mark Lawrence breamore...@yahoo.co.uk wrote: On 03/07/2014 10:27, Rita wrote: On Thu, Jul 3, 2014 at 2:54 AM, Mark Lawrence breamore...@yahoo.co.uk mailto:breamore...@yahoo.co.uk wrote: On 03/07/2014 02:17, Rita wrote: On Wed, Jul 2, 2014 at 2:46 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl__ wrote: On 2-7-2014 4:04, Rita wrote: yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. (Please don't top-post, it makes the discussion harder to follow.) On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl __ wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list [...] Why would the fact that you upgrade or remove a package, affect another developer in your group? Are you all using the same machine to develop on, with one Python installation? I think you'll have to tell us some more details about the way you work together before we can give a meaningful answer to your question. Irmen -- https://mail.python.org/__mailman/listinfo/python-list https://mail.python.org/mailman/listinfo/python-list we have a shared mount point which has our python install. we have 3 servers on one part of the campus and 2 in another part. I want to find out what packages our user base is using thats the final goal. I can figure out who is using python by writing a wrapper but not what module. -- --- Get your facts first, then you can distort them as you please.-- You can check every users's program for import statements but do you really need to, why not check what's in the site-packages folder for your python install? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence how can i get frequency of the module usage? thats the end goal. -- --- Get your facts first, then you can distort them as you please.-- Count the number of imports or count the times a given program gets run for the number of imports depending on what you mean. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list here is what I am doing now, egrep 'from|import' *.py | wc -l which is giving me that. But this does not give me the number of times the particular module gets called. I was thinking of adding a logging feature to all of my modules so every time they get called it will be written to a log file with corresponding host and username. Is there an easy way to do that? -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On Wed, Jul 2, 2014 at 2:46 PM, Irmen de Jong irmen.nos...@xs4all.nl wrote: On 2-7-2014 4:04, Rita wrote: yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. (Please don't top-post, it makes the discussion harder to follow.) On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list [...] Why would the fact that you upgrade or remove a package, affect another developer in your group? Are you all using the same machine to develop on, with one Python installation? I think you'll have to tell us some more details about the way you work together before we can give a meaningful answer to your question. Irmen -- https://mail.python.org/mailman/listinfo/python-list we have a shared mount point which has our python install. we have 3 servers on one part of the campus and 2 in another part. I want to find out what packages our user base is using thats the final goal. I can figure out who is using python by writing a wrapper but not what module. -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
general module auditing
i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list appdirs (1.3.0) colorama (0.3.1) cytoolz (0.6.1) docutils (0.11) Jinja2 (2.7.3) MarkupSafe (0.23) pip (1.5.6) py (1.4.20) pyflakes (0.8.1) Pygments (1.6) serpent (1.5, d:\projects\serpent) setuptools (4.0.1) smartypants (1.8.3) Sphinx (1.2.2) Sphinx-PyPI-upload (0.2.1) tox (1.7.1) virtualenv (1.11.6) wheel (0.23.0) You can even use pip freeze and use the resulting output as a requirements.txt file to easily install everything using pip install -r. Hope this is an idea to get you started? Irmen -- https://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
source code control and documentation
Hi, I am trying to achieve this, every time I commit to svn I automatically run hooks to test my libraries, ATM I am doing this manually cd libs PYTHONPATH=. python test_lib.py and if everything passes I do a svn commit -m 'committing code' I don't have access to my svn server so I can't create my own hooks (or can I?) Also, I would like to auto-document my library and project so, when I do a svn commit I would like to generate a sphinix-doc (and I am open to suggestions). In the source code do I need to do anything special to my Classes and Functions? class myClass(object): do I need to do anything special here? Ideally, I would like to automatically draw a UML diagram since my codebase is rather large. How do people achieve this? -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: inheriting a large python code base
when I do profiling is it possible to find out if I am spending a lot of
time in type conversion?
it seems I am not. Also, is it possible to predeclare a type in python?
Similar to C: int i=0;
On Sat, Feb 15, 2014 at 10:06 PM, Rustom Mody rustompm...@gmail.com wrote:
On Sunday, February 16, 2014 4:45:04 AM UTC+5:30, Roy Smith wrote:
In article Cameron Simpson wrote:
On 15Feb2014 12:10, Rita wrote:
i just inherited a large python code base and I would like to
optimize the
code (run faster). The application is a scientific application so I
really
don't understand the internal logic.
[...]
One thing I would keep in mind is that scientific applications
generally involve floating point math. That is subject to loss of
precision if done the wrong way.
Another thing to keep in mind is that scientific applications are often
written by {physicists, chemists, astronomers, etc} who have no clue
about how floating point math, or indeed much of anything about
computers, works :-)
Not exactly my experience.
Most of the {physicists, chemists, astronomers, etc}s I know grok
floating point better than Ive been able to do in 30 years.
They dont get much else though! -- Usually called 'Fortran programmers'.
The guy who taught me numerical analysis -- a brilliant chemist -- could
never understand why anyone should want to use anything other than Fortran.
And it was a self-evident God's truth that a variable starting with
I-N was integer Rest real.
Declarations?? Pshaw! Sissy stuff!
I *IS* an integer and no hanky-panky about it
--
https://mail.python.org/mailman/listinfo/python-list
--
--- Get your facts first, then you can distort them as you please.--
--
https://mail.python.org/mailman/listinfo/python-list
inheriting a large python code base
hi all, i just inherited a large python code base and I would like to optimize the code (run faster). The application is a scientific application so I really don't understand the internal logic. Here is what I have done so far, profile (cProfile) and was able to see where the code was taking the longest time. I was wondering if anyone else experienced something like this and how they overcame a problem such a this...what techniques have you used to optimize the code. -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: mapping objects
Thanks for the response Cameron. No amount of 'googling' could provide me with that caliber response :-) So, it seems regardless I would need a database. On Sun, Feb 2, 2014 at 1:58 AM, Cameron Simpson c...@zip.com.au wrote: On 01Feb2014 20:46, Rita rmorgan...@gmail.com wrote: I want to learn more about ORMs so I stumbled upon, SqlAlchemy. If i had a JSON document (or XML, CSV, etc.._) is it possible to convert it to a SQLAlchemy objects? I like the ability to query/filter ( http://docs.sqlalchemy.org/en/rel_0_9/orm/tutorial.html#common-filter-operators ) the data assuming I set up the proper schema. Well, not as directly as you might hope. As I recall, SQLAlchemy ORMs provide an easy way to make objects representing database entities and whose methods automatically drive the necessary SQL actions to manipulate them. On that basis, you won't get anywhere without loading up the JSON/XML/etc, parsing it for relevant information (trivial with CSV, less so for structured data like JSON or XML), and storing it in a database. To which you then point SQLAlchemy. So you're already doing the fiddly bit just to get stuff into the database. The neat filter operations you cite are actually done by special methods on the objects representing tables and columns. For example, User.name == None is done via the __eq__ method of User.name. And they return strings: bare SQL. The beauty of this is that one can write almost idiomatic python, and SQLA will generate correct SQL in the right dialect for the database backend, and with all the values correctly escaped. However, it does presume you _already_ have a backend that can be queried with SQL. Also, is this a valid way to use an ORM suite? Well, in principle sure. But SQLA won't do it for you directly. It really is for data already stored in an SQL queriable database. Your point about using SQLA's filter operations is right, _provided_ you have already loaded the original data into a database with the right schema. If you've done that work, then SQLA may well serve you well from that point on. Cheers, -- Cameron Simpson c...@zip.com.au We knew Army cadets were involved because they cut through two fences to get to the goats, and 15 feet away there was an unlocked gate. - a director of sports information in the Navy, regarding the theft of some mascots from the Naval Academy by Army rivals -- https://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
mapping objects
Hello, I want to learn more about ORMs so I stumbled upon, SqlAlchemy. If i had a JSON document (or XML, CSV, etc.._) is it possible to convert it to a SQLAlchemy objects? I like the ability to query/filter ( http://docs.sqlalchemy.org/en/rel_0_9/orm/tutorial.html#common-filter-operators) the data assuming I set up the proper schema. Also, is this a valid way to use an ORM suite? -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: data validation when creating an object
Thanks everyone for the replies. On Thu, Jan 16, 2014 at 1:36 AM, Cameron Simpson c...@zip.com.au wrote: On 16Jan2014 15:53, Ben Finney ben+pyt...@benfinney.id.au wrote: Roy Smith r...@panix.com writes: Ben Finney ben+pyt...@benfinney.id.au wrote: Who says it's frowned on to do work in the initialiser? Where are they saying it? That seems over-broad, I'd like to read the context of that advice. There are some people who advocate that C++ constructors should not do a lot of work and/or should be incapable of throwing exceptions. The pros and cons of that argument are largely C++ specific. […] Even simpler: They are mistaken in what the constructor is named, in Python. Python classes have the constructor, ‘__new__’. I would agree with advice not to do anything but allocate the resources for a new instance in the constructor. [...] Python instances have an initialiser, ‘__init__’. That function is for setting up the specific instance for later use. This is commonly over-ridden and many classes define a custom initialiser, which normally does some amount of work. I don't think ‘__init__’ is subject to the conventions of a constructor, because *‘__init__’ is not a constructor*. 99% of the time this distinction is moot. When I call ClassName(blah,...), both the constructor and initialiser are called. Informally, there's a rule of thumb that making an object (allocate, construct and initialise) shouldn't be needlessly expensive. Beyond that, what happens depends on the use patterns. This rule of thumb will be what Rita's encountered, perhaps stated without any qualification regarding what's appropriate. Cheers, -- Cameron Simpson c...@zip.com.au The problem with keeping an open mind is that my ideas all tend to fall out... - Bill Garrett garr...@cs.unc.edu -- https://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
data validation when creating an object
I would like to do some data validation when its going to a class. class Foo(object): def __init__(self): pass I know its frowned upon to do work in the __init__() method and only declarations should be there. So, should i create a function called validateData(self) inside foo? I would call the object like this x=Foo() x.validateData() Is this the preferred way? Is there a way I can run validateData() automatically, maybe put it in __init__? Or are there other techniques people use for this sort of thing? -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: data validation when creating an object
Unfortunately, I couldn't find the reference but I know I read it somewhere. Even with a selective search I wasn't able to find it. I think I read it in context of module/class test case writing. I will keep your responses in mind therefore I will put logic in __init__ for data validation. thanks again for the responses. On Wed, Jan 15, 2014 at 8:46 PM, Chris Angelico ros...@gmail.com wrote: On Thu, Jan 16, 2014 at 12:25 PM, Cameron Simpson c...@zip.com.au wrote: However, I would also have obvious validity checks in __init__ itself on the supplied values. Eg: def __init__(self, size, lifetime): if size 1: raise ValueError(size must be = 1, received: %r % (size,)) if lifetime = 0: raise ValueError(lifetime must be 0, received: %r % (lifetime,)) Trivial, fast. Fails early. Note that the exception reports the receive value; very handy for simple errors like passing utterly the wrong thing (eg a filename when you wanted a counter, or something like that). With code like this, passing a filename as the size will raise TypeError on Py3: size = test.txt size 1 Traceback (most recent call last): File stdin, line 1, in module TypeError: unorderable types: str() int() Yet another advantage of Py3 :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
general ConfigParser question
Hi, I was wondering if the default ConfigParser can handle multi line strings (especially in the relate section) For example, if i have system.ini [Global] memory = 1024 [Process A] command = sleep arguments = 100 [Process B] command = nslookup arguments = hostA output = data [Process C] command = checksum arguments = data [Relate] data=parent process A child process B Parent process B child process C --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
python process accounting
Hi, I was wondering if it possible to write a python wrapper which will account my processes. I would like to account for all the children processes (fork) by looking at their /proc/pid info. Such as memory, io, open files, stats. So, instead of me running /bin/sleep 10, i would like to run it as pywrap.py /bin/sleep 10 and it will do an exec /bin/sleep 10 and do a periodic snapshot for whats in /proc/pid/stats. -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: python process accounting
Exactly, its a place holder measure. Currently, I have a shell program like this. #!/usr/bin/env bash exec $@ Any thoughts how I can put python in there? On Tue, Apr 30, 2013 at 1:54 PM, Robert Kern robert.k...@gmail.com wrote: On 2013-04-30 17:38, Dave Angel wrote: On 04/30/2013 12:25 PM, Rita wrote: Hi, I was wondering if it possible to write a python wrapper which will account my processes. I would like to account for all the children processes (fork) by looking at their /proc/pid info. Such as memory, io, open files, stats. So, instead of me running /bin/sleep 10, i would like to run it as pywrap.py /bin/sleep 10 and it will do an exec /bin/sleep 10 and do a periodic snapshot for whats in /proc/pid/stats. I only understood a quarter of that. But if you want to sleep, why not just call time.sleep ? I think that was just a placeholder example, not the program he actually wants to measure. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/**mailman/listinfo/python-listhttp://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: Python scheduler
Thanks for the replies. Its also important for me to have asynchronous abilities. I will be running some programs which will be taking much longer (1-2 minutes) and I don't want others to be blocked. On Thu, Feb 21, 2013 at 2:35 AM, Chris Rebert c...@rebertia.com wrote: On Wed, Feb 20, 2013 at 8:04 PM, Rita rmorgan...@gmail.com wrote: Hello, Here is what I am trying to do. (Currently, I am doing this in cron but i need much more granularity). I am trying to run program every 20 secs and loop forever. I have several of these types of processes, some should run every 5 mins, 10 secs, 20 secs, 1 min and so forth. I was wondering what is the best way to do this? I've had a good experience with APScheduler (http://pythonhosted.org/APScheduler/ ), although my particular use-case was trivial, but it totally supports periodic scheduling. Cheers, Chris -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: Python scheduler
Thanks. This is what I was looking for! On Thu, Feb 21, 2013 at 9:48 AM, Adam Tauno Williams awill...@whitemice.org wrote: On Wed, 2013-02-20 at 23:04 -0500, Rita wrote: Here is what I am trying to do. (Currently, I am doing this in cron but i need much more granularity). I am trying to run program every 20 secs and loop forever. I have several of these types of processes, some should run every 5 mins, 10 secs, 20 secs, 1 min and so forth. I was wondering what is the best way to do this? If you need a scheduler - use a scheduler! :) https://pypi.python.org/pypi/APScheduler/ Works great, and very flexible. -- Adam Tauno Williams GPG D95ED383 Systems Administrator, Python Developer, LPI / NCLA -- http://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
children process
hi, I currently have a bash wrapper which executes a program, something like this #!/usr/bin/env bash export LD_LIBRARY_PATH=/foo:$LD_LIBRARY_PATH exec $@ I would like to have a python process which will do process accounting for all children, so if a process starts, I would like to get all the children's memory (/proc/ALL Children/stat). Is there a way to do this? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
building python from source
For those building python from source what are some tests you do to make sure the compilation and installation is up to standard. For instance here are some thing I do: Tk functionality sqlite module Python is compiled with shared object (important for wsgi) Proper preloading of python libraries (set the proper -rpath flag for gcc) Any others? The people who manage distributions what do they check with for regression tests? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Urwid 1.1.0 - Usability and Documentation
+1 great library. I suppose it will never be in the standard library? :p On Wed, Oct 24, 2012 at 5:08 AM, Jean-Michel Pichavant jeanmic...@sequans.com wrote: - Original Message - Announcing Urwid 1.1.0 -- Urwid home page: http://excess.org/urwid/ Manual: http://excess.org/urwid/docs/ Package: http://pypi.python.org/pypi/urwid/1.1.0 [snip] Hi, I just had a look at it. Amazing job. JM -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. -- http://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
unittest for system testing
Hi, Currently, I use a shell script to test how my system behaves before I deploy an application. For instance, I check if fileA, fileB, and fileC exist and if they do I go and start up my application. This works great BUT I would like to use python and in particular unittest module to test my system and then deploy my app. I understand unittest is for functional testing but I think this too would be a case for it. Any thoughts? I am not looking for code in particular but just some ideas on how to use python better in situations like this. -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: unittest for system testing
thanks. I suppose I would need a simple example from one of these libraries. ( i typed too soon for , no code needed ) On Wed, Oct 17, 2012 at 8:49 PM, Mark Lawrence breamore...@yahoo.co.ukwrote: On 18/10/2012 01:22, Rita wrote: Hi, Currently, I use a shell script to test how my system behaves before I deploy an application. For instance, I check if fileA, fileB, and fileC exist and if they do I go and start up my application. This works great BUT I would like to use python and in particular unittest module to test my system and then deploy my app. I understand unittest is for functional testing but I think this too would be a case for it. Any thoughts? I am not looking for code in particular but just some ideas on how to use python better in situations like this. Plenty of options here http://wiki.python.org/moin/** PythonTestingToolsTaxonomyhttp://wiki.python.org/moin/PythonTestingToolsTaxonomyand an active mailing list that I read via gmane.comp.python.testing. **general -- Cheers. Mark Lawrence. -- http://mail.python.org/**mailman/listinfo/python-listhttp://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: best way to handle this in Python
Thats an interesting data structure Dennis. I will actually be running this
type of query many times preferable in an ad-hoc environment. That makes it
tough for sqlite3 since there will be several hundred thousand tuples.
On Fri, Jul 20, 2012 at 12:18 AM, Dennis Lee Bieber
wlfr...@ix.netcom.comwrote:
{NOTE: preferences for comp.lang.python are to follow the RFC on
netiquette -- that is, post comments /under/ quoted material, trimming
what is not relevant... I've restructured this reply to match}
On Thu, 19 Jul 2012 21:28:12 -0400, Rita rmorgan...@gmail.com
declaimed the following in gmane.comp.python.general:
On Thu, Jul 19, 2012 at 8:52 PM, Dave Angel d...@davea.name wrote:
On 07/19/2012 07:51 PM, Rita wrote:
Hello,
I have data in many files (/data/year/month/day/) which are named
like
YearMonthDayHourMinute.gz.
I would like to build a data structure which can easily handle
querying
the
data. So for example, if I want to query data from 3 weeks ago till
today,
i can do it rather quickly.
each YearMonthDayHourMinute.gz file look like this and they are
about 4to
6kb
red 34
green 44
blue 88
orange 4
black 3
while 153
I would like to query them so I can generate a plot rather quickly
but
not
sure what is the best way to do this.
What part of your code is giving you difficulty? You didn't post any
code. You don't specify the OS, nor version of your Python, nor what
other programs you expect to use along with Python.
Using linux 2.6.31; Python 2.7.3.
I am not necessary looking for code just a pythonic way of doing it.
Eventually, I would like to graph the data using matplotlib
Which doesn't really answer the question. After all, since the
source data is already in date/time-stamped files, a simple, sorted,
glob of files within a desired span would answer the requirement.
But -- it would mean that you reparse the files for each processing
run.
An alternative would be to run a pre-processor that parses the
files
into, say, an SQLite3 database (and which can determine, from the
highest datetime entry in the database, which /new/ files need to be
parsed on subsequent runs). Then do the query/plotting from a second
program which retrieves data from the database.
But if this is a process that only needs to be run once, or at rare
intervals, maybe you only need to parse the files into an in-memory data
structure... Say a list of tuples of the form:
[ (datetime, {color: value, color2: value2, ...}),
(datetime2,
...) ]
--
Wulfraed Dennis Lee Bieber AF6VN
wlfr...@ix.netcom.comHTTP://wlfraed.home.netcom.com/
--
http://mail.python.org/mailman/listinfo/python-list
--
--- Get your facts first, then you can distort them as you please.--
--
http://mail.python.org/mailman/listinfo/python-list
best way to handle this in Python
Hello, I have data in many files (/data/year/month/day/) which are named like YearMonthDayHourMinute.gz. I would like to build a data structure which can easily handle querying the data. So for example, if I want to query data from 3 weeks ago till today, i can do it rather quickly. each YearMonthDayHourMinute.gz file look like this and they are about 4to 6kb red 34 green 44 blue 88 orange 4 black 3 while 153 I would like to query them so I can generate a plot rather quickly but not sure what is the best way to do this. -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: best way to handle this in Python
Using linux 2.6.31; Python 2.7.3. I am not necessary looking for code just a pythonic way of doing it. Eventually, I would like to graph the data using matplotlib On Thu, Jul 19, 2012 at 8:52 PM, Dave Angel d...@davea.name wrote: On 07/19/2012 07:51 PM, Rita wrote: Hello, I have data in many files (/data/year/month/day/) which are named like YearMonthDayHourMinute.gz. I would like to build a data structure which can easily handle querying the data. So for example, if I want to query data from 3 weeks ago till today, i can do it rather quickly. each YearMonthDayHourMinute.gz file look like this and they are about 4to 6kb red 34 green 44 blue 88 orange 4 black 3 while 153 I would like to query them so I can generate a plot rather quickly but not sure what is the best way to do this. What part of your code is giving you difficulty? You didn't post any code. You don't specify the OS, nor version of your Python, nor what other programs you expect to use along with Python. -- DaveA -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
recompiling standard modules
I noticed that active state python Tk inter isnt compiled with --enable-threads therefore I would like to recompile the module with a new version of TCL/TK which is compiled with threads. How can I do this? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
other languages API to python
Hello, A vendor provided a C, C++ and Java API for a application. They dont support python so I would like to create a library for it. My question is, how hard/easy would it be to create something like this? Is there a simple HOWTO or examples I can follow? Can someone shed home light on this? TIA -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
standard module xpath
Hello, Does python ship with xpath query language support? Certainly, I can use lxml but I prefer something in the standard modules list. -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
bash/shell to python
Hello, I currently build a lot of interfaces/wrappers to other applications using bash/shell. One short coming for it is it lacks a good method to handle arguments so I switched to python a while ago to use 'argparse' module. Its a great complement to subprocess module. I was wondering if there is a generic framework people follow to build python scripts which are replacing shell scripts? Is there a guide or a template to follow? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
compiling Tkinter
Hello, I understand Tkinter is part of the python distribution but for me it always fails when I try to load the module. I get: import Tkinter ... import _tkinter # if this fails your Python may not be configured for Tk ImportError: No module named _tkinter So, here is how I am compiling tcl/tk and Python please let me know if you see any issues with this. assuming, my prefix is /tmp/testdir wget http://prdownloads.sourceforge.net/tcl/tcl8.5.11-src.tar.gz wget http://prdownloads.sourceforge.net/tcl/tk8.5.11-src.tar.gz tar -xzpf tcl8.5.11-src.tar.gz tar -xzpf tk8.5.11-src.tar.gz #compile tcl cd tcl8.5.11/unix/ ./configure --prefix=/tmp/testdir --enable-64bit --enable-threads make make test make install #compile tk cd tk8.5.11/unix/ ./configure --prefix=/tmp/testdir --enable-threads --enable-64-bit --with-tcl=/tmp/testdir/lib make make install #compile python wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz tar -xzpvf Python-2.7.3.tgz cd Python-2.7.3/ ./configure --prefix=/tmp/testdir --enable-shared LDFLAGS=-Wl,-rpath,/tmp/testdir/lib Any thoughts? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
python library to generate restructured text
Hello, I am using sqlite3 modules to get data out of a table. I would like to dump the table into a restructured-text (docutil) format. I was wondering if there was a library which would accomplish this. -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: standalone python web server
yes, I would like to use a framework. I like the twisted method the user posted. Are there any examples of it using a framework, get/post, etc..? On Thu, Feb 9, 2012 at 6:28 AM, Thomas Bach thb...@students.uni-mainz.dewrote: Rita rmorgan...@gmail.com writes: I am building a small intranet website and I would like to use Python. I was wondering if there was a easy and medium performance python based web server available. Are you going to use a framework? Most of these ship with a light web server implementation… You probably want to check these out too… I got a pretty robust setup via + nginx, + supervisord, + the web server shipped with Pyramid If you want to use the WSGI standard I'd recommend to read http://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/deployment.html This document describes several possible set ups to serve an WSGI application… Regards, vince -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
standalone python web server
I am building a small intranet website and I would like to use Python. I was wondering if there was a easy and medium performance python based web server available. I would like to run it on port :8080 since I wont have root access also I prefer something easy to deploy meaning I would like to move the server from one physical host to another without too much fuss. Currently, I am using flask (development server) and everything is ok but the performance is really bad. -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
working with a large file
I have a large file, 18gb uncompressed, and I would like to know what is the preferred method to read this file for random access. I have several processes reading the file which different calculate arguments. My server has 64gb of memory. Not sure what is the preferred way to do this? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
code generation
Hello, This isn't much of a python question but a general algorithm question. I plan to input the following string and I would like to generate something like this. input: a-(b,c)-d output: parent a, child b c parent b c child d Are there any libraries or tools which will help me evaluate items like this better? I am mostly looking for syntax and sanity checking. Any thoughts? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
complex data structures in python
Hello, I was wondering if anyone has any documentation/recipes for implementing complex data structures. For instance, if you had a dictionary with a list inside a list inside a set. -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
PyCon
Hello, I noticed the old presentations at PyCon aren't avaliable anymore. http://us.pycon.org/2010/ http://us.pycon.org/2009/ Does anyone know where else I can get these presentations? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: parsing a file for analysis
Thanks Andrea. I was thinking that too but I was wondering if there were any other clever ways of doing this. I also though, I can build a filesystem structure depending on the __time. So, for January 01, 2011. I would create /tmp/data/20110101/data . This way I can have a fast index of the data. And next time I read thru this file, I can skip all of Jan 01, 2011 On Sat, Feb 26, 2011 at 10:29 AM, Andrea Crotti andrea.crott...@gmail.comwrote: Il giorno 26/feb/2011, alle ore 06.45, Rita ha scritto: I have a large text (4GB) which I am parsing. I am reading the file to collect stats on certain items. My approach has been simple, for row in open(file): if INFO in row: line=row.split() user=line[0] host=line[1] __time=line[2] ... I was wondering if there is a framework or a better algorithm to read such as large file and collect it stats according to content. Also, are there any libraries, data structures or functions which can be helpful? I was told about 'collections' container. Here are some stats I am trying to get: *Number of unique users *Break down each user's visit according to time, t0 to t1 *what user came from what host. *what time had the most users? (There are about 15 different things I want to query) I understand most of these are redundant but it would be nice to have a framework or even a object oriented way of doing this instead of loading it into a database. Any thoughts or ideas? Not an expert, but maybe it might be good to push the data into a database, and then you can tweak the DBMS and write smart queries to get all the statistics you want from it. It might take a while (maybe with regexp splitting is faster) but it's done only once and then you work with DB tools. -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: parsing a file for analysis
Yes, Yes :-). I was using awk to do all of this. It does work but I find
myself repeating reading the same data because awk does not support complex
data structures. Plus the code is getting ugly.
I was told about Orange (http://orange.biolab.si/). Does anyone have
experience with it?
On Sat, Feb 26, 2011 at 10:53 AM, Martin Gregorie
martin@address-in-sig.invalid wrote:
On Sat, 26 Feb 2011 16:29:54 +0100, Andrea Crotti wrote:
Il giorno 26/feb/2011, alle ore 06.45, Rita ha scritto:
I have a large text (4GB) which I am parsing.
I am reading the file to collect stats on certain items.
My approach has been simple,
for row in open(file):
if INFO in row:
line=row.split()
user=line[0]
host=line[1]
__time=line[2]
...
I was wondering if there is a framework or a better algorithm to read
such as large file and collect it stats according to content. Also, are
there any libraries, data structures or functions which can be helpful?
I was told about 'collections' container. Here are some stats I am
trying to get:
*Number of unique users
*Break down each user's visit according to time, t0 to t1 *what user
came from what host.
*what time had the most users?
(There are about 15 different things I want to query)
I understand most of these are redundant but it would be nice to have a
framework or even a object oriented way of doing this instead of
loading it into a database.
Any thoughts or ideas?
Not an expert, but maybe it might be good to push the data into a
database, and then you can tweak the DBMS and write smart queries to get
all the statistics you want from it.
It might take a while (maybe with regexp splitting is faster) but it's
done only once and then you work with DB tools.
This is the sort of job that is best done with awk.
Awk processes a text file line by line, automatically splitting each line
into an array of words. It uses regexes to recognise lines and trigger
actions on them. For example, building a list of visitors: assume there's
a line containing username logged on, you could build a list of users
and count their visits with this statement:
/logged on/ { user[$1] += 1 }
where the regex, /logged on/, triggers the action, in curly brackets, for
each line it matches. $1 is a symbol for the first word in the line.
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
--
http://mail.python.org/mailman/listinfo/python-list
--
--- Get your facts first, then you can distort them as you please.--
--
http://mail.python.org/mailman/listinfo/python-list
Re: subprocess pipe question
The results look right! I did a rather large test and the checksum passed. I will hold off any speed ups as you suggested. On Wed, Feb 23, 2011 at 8:37 PM, Rob Williscroft r...@rtw.me.uk wrote: Rita wrote in news:AANLkTi=88dcpm_kqrs2g620obsnxz0majubfwpeme...@mail.gmail.com in gmane.comp.python.general: [Top post relocated] On Tue, Feb 22, 2011 at 7:57 PM, Rob Williscroft r...@rtw.me.uk wrote: Rita wrote in news:AANLkTi=w95gxosc1tkt2bntgjqys1cbmdnojhokq4...@mail.gmail.com in gmane.comp.python.general: When using wait() it works a bit better but not consistent def run(cmd): p=subprocess.Popen(cmd,stdout=subprocess.PIPE) rc=p.wait() print rc return p.stdout When the output of cmd is a small ascii file it works perfectly fine, but when the file is large (more than 2MB) the process just waits for ever (I am guessing its blocking?). Your OS has supplied a pipe buffer of 2MB, its full and the prossess is waiting until you read something from the pipe (i.e. p.stdout.read()). When I use the communicate call it works perfectly but my process is consuming way too much memory. Is there a better way to get my return code consistently efficiently and not take up so much memory? If you don't need the processes output then don't use the PIPE argument. If you do you will need to read from p.stdout until the process is complete, then get the return code: while True: buf = p.stdout.read( 1024 ) # do somthing with buf if len( buf ) 1024: break rc = p.wait() For extra points is there a way to speed up the p.stdout.read(bufsize) ? Its unlikely that is the problem, most likely you are reading all of the 2MB OS pipe buffer and then having to wait for the programme you are calling to produce some more output. Before trying to speed things up be sure to test with real work being done to the output you recieve. If its still slow and you have multiple cores/processors then you will need to put you read loop in a thread to speed it up. -- Rob. -- http://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
parsing a file for analysis
I have a large text (4GB) which I am parsing. I am reading the file to collect stats on certain items. My approach has been simple, for row in open(file): if INFO in row: line=row.split() user=line[0] host=line[1] __time=line[2] ... I was wondering if there is a framework or a better algorithm to read such as large file and collect it stats according to content. Also, are there any libraries, data structures or functions which can be helpful? I was told about 'collections' container. Here are some stats I am trying to get: *Number of unique users *Break down each user's visit according to time, t0 to t1 *what user came from what host. *what time had the most users? (There are about 15 different things I want to query) I understand most of these are redundant but it would be nice to have a framework or even a object oriented way of doing this instead of loading it into a database. Any thoughts or ideas? --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess pipe question
This is a good solution thanks. You should wiki this somewhere. For extra points is there a way to speed up the p.stdout.read(bufsize) ? On Tue, Feb 22, 2011 at 7:57 PM, Rob Williscroft r...@rtw.me.uk wrote: Rita wrote in news:AANLkTi=w95gxosc1tkt2bntgjqys1cbmdnojhokq4...@mail.gmail.com in gmane.comp.python.general: When using wait() it works a bit better but not consistent def run(cmd): p=subprocess.Popen(cmd,stdout=subprocess.PIPE) rc=p.wait() print rc return p.stdout When the output of cmd is a small ascii file it works perfectly fine, but when the file is large (more than 2MB) the process just waits for ever (I am guessing its blocking?). Your OS has supplied a pipe buffer of 2MB, its full and the prossess is waiting until you read something from the pipe (i.e. p.stdout.read()). When I use the communicate call it works perfectly but my process is consuming way too much memory. Is there a better way to get my return code consistently efficiently and not take up so much memory? If you don't need the processes output then don't use the PIPE argument. If you do you will need to read from p.stdout until the process is complete, then get the return code: while True: buf = p.stdout.read( 1024 ) # do somthing with buf if len( buf ) 1024: break rc = p.wait() Rob. -- http://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
subprocess pipe question
I have a process like this, def run(cmd): #cmd=a process which writes a lot of data. Binary/ASCII data p=subprocess.Popen(cmd,stdout=subprocess.PIPE) I would like to get cmd's return code so I am doing this, def run(cmd): p=subprocess.Popen(cmd,stdout=subprocess.PIPE) rc=p.poll() print rc # and I get 'None' regardless of what cmd gives me (0 thru 255 are valid return codes) return p.stdout When using wait() it works a bit better but not consistent def run(cmd): p=subprocess.Popen(cmd,stdout=subprocess.PIPE) rc=p.wait() print rc return p.stdout When the output of cmd is a small ascii file it works perfectly fine, but when the file is large (more than 2MB) the process just waits for ever (I am guessing its blocking?). When I use the communicate call it works perfectly but my process is consuming way too much memory. Is there a better way to get my return code consistently efficiently and not take up so much memory? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess pipe question
Thanks everyone for your replies. Chris, Unfortunately, I can't redirect the output to a file because there are other processes which use this processes output as stdin. Rob, I will give this a try. On Tue, Feb 22, 2011 at 7:48 PM, Chris Kaynor ckay...@zindagigames.comwrote: On Tue, Feb 22, 2011 at 3:44 PM, Rita rmorgan...@gmail.com wrote: I have a process like this, def run(cmd): #cmd=a process which writes a lot of data. Binary/ASCII data p=subprocess.Popen(cmd,stdout=subprocess.PIPE) I would like to get cmd's return code so I am doing this, def run(cmd): p=subprocess.Popen(cmd,stdout=subprocess.PIPE) rc=p.poll() print rc # and I get 'None' regardless of what cmd gives me (0 thru 255 are valid return codes) return p.stdout Here, you are accessing the return code, however the process has not exited yet, and thus you get None. When using wait() it works a bit better but not consistent def run(cmd): p=subprocess.Popen(cmd,stdout=subprocess.PIPE) rc=p.wait() print rc return p.stdout When the output of cmd is a small ascii file it works perfectly fine, but when the file is large (more than 2MB) the process just waits for ever (I am guessing its blocking?). When I use the communicate call it works perfectly but my process is consuming way too much memory. Here, you wait for the process to exit, with wait returning the exit code. As you have waited for the process, it has exited and thus you get the return code. The issue here is that the stdout pipe has a limited size, and the process will block once the pipe becomes full. You are entering a dead-lock as the pipe is full, and thus the sub process is blocking waiting for room to write more output. As nothing is reading that output, that will never happen. It works fine on smaller output as you don't hit the limit. See the warning on http://docs.python.org/library/subprocess.html#popen-objects under wait. .communicate works as it reads the output, however it eats up memory as all of the data has to be buffered in memory. A possible work around would be to redirect the output into a file rather than to a pipe (see the comments about stdin, stdout and stderr at http://docs.python.org/library/subprocess.html#using-the-subprocess-module). In this way, the data will be buffered to a file rather than memory. Is there a better way to get my return code consistently efficiently and not take up so much memory? -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- http://mail.python.org/mailman/listinfo/python-list
Converting from shell to python
Hello,
I have been using shell for a long time and I decided to learn python
recently. So far I am liking it a lot especially the argparse module which
makes my programs more professional like.
Currently, I am rewriting my bash scripts to python so I came across a
subprocess and environment problem.
My bash script looks like this,
#!/usr/bin/env bash
export JAVA_HOME=/opt/java
export PROG=dbconn
export JAVA_ARGS=-Xmx16g
export ARGS=-out outdata.dat
$JAVA_HOME $JAVA_ARGS $PROG $ARGS
To convert this into python I did something like this (which isn't working
properly)
#!/usr/bin/env python
import subprocess,os
def execute():
try:
cmd=$JAVA_HOME $JAVA_ARGS $PROG $ARGS
cmd=cmd.split()
p=subprocess.Popen([cmd],env={JAVA_HOME:/opt/java,PROG:dbconn,JAVA_ARGS:-Xmx16g,ARGS:-out
outdata.dat},
stdout=subprocess.PIPE)
except OSError, e:
print sys.stderr, Execution failed: ,e
return p.stdout
So, I was wondering if this is the correct way of doing it? or is there an
alternative?
--
--- Get your facts first, then you can distort them as you please.--
--
http://mail.python.org/mailman/listinfo/python-list
