C replacement for Queue module
I'm working on an application that makes heavy use of Queue objects in a multithreaded environment. By "heavy" I mean "millions of calls to put and get, constituting ~20% of the app's run time." The profiler thinks that a significant amount of time is spent in this code -- not just a consumer waiting for a producer, but actual _empty, notify calls, etc. Would it be worth the time to write a CQueue module with pthread_cond instead of Python Condition objects, etc? I don't really have a gut feeling for how much bang-for-the-loc C optimization might give: twice as fast? 5x as fast? thanks, -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: C replacement for Queue module
[EMAIL PROTECTED] wrote: > does collections.deque have a blocking popleft()? If not, it's not very > suitable to replace Queue.Queue. It does not. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: gnuplot on Canvas widget
Blues wrote: > I have used two great models - Tkinter and Gnuplot.py - for a while. I > can display an image on a Canvas widget using Tkinter and I can also > generate a gnuplot from Python on the fly in a separate window. Does > anyone know how to display such a gnuplot on the Canvas widget with an > image in it? Thanks. >From my experience, the Gnuplot module isn't designed to be used in "headless" mode -- it can save to the usual formats, but you have to render everything in an x11 window interactively first. It might not be hard to modify this, though. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: pexpect question....
[EMAIL PROTECTED] wrote: > Currently, I am spawning a new thread > that just does pexpect_spawned_child.close(wait=1). It seems to work in > some cases but the child process is occassionally getting deadlocked. I think your only cross-platform option will be to fix the child process to die nicely instead of trying to find better ways kill it. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: threads and sleep?
Jeffrey Maitland wrote: > The problem I have is I had an application > (wrote/co-wrote) that has a long run time dependant on some variables > passed to it (mainly accuracy variables, the more accurate the longer > the run time - makes sense). However in the hopes to speed it up I > decided to write a threaded version of the program to try and speed > it up. How ever what I am noticing is that the threaded version is > taking as long possibly longer to run. The thing is the threaded > version is running on an 8 ia-64 proccessor system and it seems to > only be using 2 or 3 porcessors at about 30% (fluxiates). My guess is > that 6 threads are running they are using 30% sprox each of a 2 given > CPUS. In many ways, Python is an incredibly bad choice for deeply multithreaded applications. One big problem is the global interpreter lock; no matter how many CPUs you have, only one will run python code at a time. (Many people who don't run on multiple CPUs anyway try to wave this off as a non-problem, or at least worth the tradeoff in terms of a simpler C API, but with multicore processors coming from every direction I think the "let's pretend we don't have a problem" approach may not work much longer.) If the GIL isn't an issue (and in your case it clearly is), you'll quickly find that there's little support for debugging multithreaded applications, and even less for profiling. Sometimes running multiple processes is an acceptable workaround; if not, good luck with the rewrite in Java or something else with real thread support. (IIRC Jython doesn't have a GIL; that might be an option too.) Python is a great tool but if you really need good threading support you will have to look elsewhere. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: threads and sleep?
Peter Hansen wrote: > Jeffrey Maitland wrote: > > I was hoping that python would allow for the cpu threading such in > > Java etc.. but I guess not. (from the answers,and other findings) I > > guess I will have to write this part of the code in something such as > > java or c or something that allows for it then I can either wrap it in > > python or avoid python for this part of the app. > > Or investigate the use of Irmen's Pyro package and how it could let you > almost transparently move your code to a *multi-process* architecture Unless you're doing anything that would require distributed locking. Many if not most such projects do, which is why almost everyone prefers to use threads on an SMP machine instead of splitting it across multiple smaller boxes. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: threads and sleep?
Peter Hansen wrote: > Jonathan Ellis wrote: > > Peter Hansen wrote: > >>Or investigate the use of Irmen's Pyro package and how it could let you > >>almost transparently move your code to a *multi-process* architecture > > > > Unless you're doing anything that would require distributed locking. > > Many if not most such projects do, which is why almost everyone prefers > > to use threads on an SMP machine instead of splitting it across > > multiple smaller boxes. > > I can't address the issue of whether or not "most" such projects require > distributed locking, because I'm not familiar with more than half of > such projects, as you appear to be. Your sarcasm is cute, I suppose, but think about it for a minute. If the opposite of what I assert is true, why would even the mainstream press be running articles along the lines of "multicore CPUs mean programming will get tougher because locking is hard to get right and you can't just scale by relying on the cpu to run your one thread/process really fast anymore." http://www.gotw.ca/publications/concurrency-ddj.htm for one example. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: calling python procedures from tcl using tclpython
Jeff Hobbs wrote:
> chand wrote:
> > can anyone help me how to provide the info about the python file
> > procedure in the tcl script which uses tclpython i.e., is there a way
> > to import that .py file procedure in the tcl script
>
> >>>currently I have wriiten this tcl code which is not working
> >>>
> >>>package require tclpython
> >>>set interpreter [python::interp new]
> >>>$interpreter eval {def test_function(): arg1,arg2} ;
> >>>python::interp delete $interpreter
>
> You would call 'import' in the python interpreter, like so:
> $interpreter eval { import testfile }
> assuming it's on the module search path. Look in the python
> docs about Modules to get all the info you need.
Actually, both your import and the original "def" problem need to use
exec instead of eval. Eval works with expressions; for statements you
need exec.
I blogged a brief example of tclpython over here:
http://spyced.blogspot.com/2005/06/tale-of-wiki-diff-implementation.html
-Jonathan
--
http://mail.python.org/mailman/listinfo/python-list
Re: Yet Another Python Web Programming Question
Daniel Bickett wrote: > He would read the documentation of Nevow, Zope, and Quixote, and would > find none of them to his liking because: > > * They had a learning curve, and he was not at all interested, being > eager to fulfill his new idea for the web app. It was his opinion that > web programming should feel no different from desktop programming. If you're coming from a PHP background, you'll find Spyce's learning curve even shallower. http://spyce.sourceforge.net -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
ANN: Spyce 2.0.3
Spyce 2.0.3 released Spyce is a python web application server, combining features of popular frameworks such as ASP.NET and JSP with Pythonic elegance. Spyce may be deployed as a standalone server, proxied behind Apache, under mod_python, FastCGI, or CGI. Spyce 2.0 includes features unique to Python web frameworks such as Active Handlers - reusable components without the leaky abstractions seen in ASP.NET et al. - http://spyce.sourceforge.net/docs/doc-lang_handlers.html Active Tag compiler - http://spyce.sourceforge.net/docs/doc-tag_new2.html 2.0.3 is a bugfix and documentation-improvement release. The installation section of the manual has received particular attention. There is also the new section on starting your first project (http://spyce.sourceforge.net/docs/doc-conf_next.html), which answers the FAQ, "how do I organize my Spyce files." Demos and downloads are available at http://spyce.sourceforge.net/. Changelog is at http://svn-hosting.com/svn/spyce/trunk/spyce/CHANGES. Jonathan Ellis http://spyced.blogspot.com -- http://mail.python.org/mailman/listinfo/python-list
datagram queue length
I seem to be running into a limit of 64 queued datagrams. This isn't a
data buffer size; varying the size of the datagram makes no difference
in the observed queue size. If more datagrams are sent before some are
read, they are silently dropped. (By "silently," I mean, "tcpdump
doesn't record these as dropped packets.") This is a problem because
while my consumer can handle the overall load easily, the requests
often come in large bursts.
This only happens when the sending and receiving processes are on
different machines, btw.
Can anyone tell me where this magic 64 number comes from, so I can
increase it?
Illustration follows.
-Jonathan
#
# start this, then immediately start the other
# _on another machine_
import socket, time
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('', 3001))
time.sleep(5)
while True:
data, client_addr = sock.recvfrom(8192)
print data
#
import socket
for i in range(200):
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto('a' * 100, 0, ('***other machine ip***', 3001))
sock.close()
--
http://mail.python.org/mailman/listinfo/python-list
Re: Django Vs Rails
James wrote: > I actually like the framework to reflect on my database. I am more of a > visual person. I have tools for all my favorite databases that allow me > to get a glance of ER diagrams and I would rather develop my data > models in these tools rather than in code. Further more I rather like > the idea of parsimonious use of code (which is probably why I use > Python in the first place) and do not really like manually specifying > data schemas in code as much as possible. > > Is some familiar with a Python Framework that builds by reflection. PyDO (http://skunkweb.sourceforge.net/pydo2.html) is a Python ORM tool that does this well (*cough* better than sqlobject *cough*). -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: web scrapping - POST and auto-login
"james hal-pc.org" wrote: >I'm trying to update the WEP key on a wireless router via script and > email the results to myself. this will be run once a week. Look up Mechanize (http://wwwsearch.sourceforge.net/mechanize/) or the more low-level ClientForm by the same author. This will be _much_ easier than doing it by hand. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: cookie lib policy how-tp?
Riko Wichmann wrote:
>
> When I use opera to access this page by hand and look at the sources,
I
> see the full sources when letting opera identify itself as MSIE 6.0.
> When using Mozilla 5.0 I get the same in-complete source file as with
> python.
Sounds like your first step should be to identify yourself as IE.
opener = urllib2.build_opener(...)
opener.addheaders = [("User-Agent", "whatever IE calls itself these
days")]
-Jonathan
--
http://mail.python.org/mailman/listinfo/python-list
Re: Visual Python, really "Visual"?
Luis M. Gonzalez wrote: > The IDEs you've been looking at have no visual GUI designers. > For that, you can check Boa Constructor or PythonCard. These two are > based on the wxPython toolkits. There are other commercial IDEs based > on QT but I cannot comment on these cause I've never used them. > IMHO, Komodo or WingIde don't offer anything that can't be found in > free products. You're poorly informed. Komodo, for instance, does indeed offer a GUI designer for tkinter. Even where functionality is similar, Komodo and Wing are both ridiculously more polished than the free alternatives. For some people that's not important. For people that use their tools 40 hours a week or more, it often is. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: Wingide is a beautiful application
James wrote: > >> I haven't used an IDE in a long time but gave wing ide a try because > >> I wanted the same development platform on Linux and Windows. > Then you owe it to yourself to also try SPE, PyDev and Boa Constructor > (got off to a slow start, but it looks promising now). All are free, > open source, cross platform, native look and feel and support more or > less the features you list. In my experience Boa Constructor isn't worth bothering with. It's far too bugy for practical use. > Two minor peeves about WingIDE. > 1.) Not native look and feel. Well... GTK is as native as anything else, on Linux. Even on Windows GTK apps seem to be spreading; I've been using GIMP and GAIM long enough that I guess "not quite native" doesn't bug me anymore. > 2.) Auto List members implementation is great. But what about call > tips? Just as important and every other Python IDE has it. Wing shows calltip info in the Source Assistant panel. (Pro version only, IIRC.) -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: Need a compelling argument to use Django instead of Rails
Jaroslaw Zabiello wrote: > On Mon, 24 Jul 2006 12:23:12 +0100, Steve Holden wrote: > > > > The impression I get is that Rails is relatively inflexible > > on database schemas, > > Django has the same problem. E.g. both Django ORM and ActiveRecord cannot > work with complex primary keys. But for Rails there is a solution for even > very strange created databases: rBatis > (http://jutopia.tirsen.com/articles/tag/rbatis) it is Ruby port of java > Ibatis (http://ibatis.apache.org/) The only commits to the rBatis svn repository happened on May 20. Looks like it's going nowhere fast to me. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: pass parameters in SPYCE
kepioo wrote: > Hi all, > > I started to use the so good spyce server. I manage to do all the > basics, however, I still block on one problem : > > How can I pass parameters to a spy page : example > > I have an index page : > > link1 > link2 > link3 > link4 > > I want all theses html links to point to process.spy with the value of > the link as parameter ( equivalent to call process.spy -link1 ). > > Is this feasible with html links or I have to use forms and replace my > links with buttons? You mean, a href=process.spy?link=1 ? Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Diamanda Wiki and MyghtyBoard Forum Test 1
[EMAIL PROTECTED] wrote: > Diamanda Wiki and MyghtyBoard Forum Test 1 > > Diamanda is a wiki django application and Myghty Board is a bulletin > board application. Both written in Django >= 0.95. Might want to re-think your bboard app's name; people might think it runs on Myghty. :) -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: Get my airlines boarding pass
KenAggie wrote: > I posted it on activestate already ... sorry. I would like for the > geniuses here to use it and improve upon it so here is the activestate > post URL: > > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496790 Whoa. Check out BeautifulSoup -- you will never write HTMLParser-based screen scrapers again. :) -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: Very nice python IDE (windows only)
ago wrote: > I have just discovered Python Scripter by Kiriakos Vlahos and it was a > pleasant surprise. I thought that it deserved to be signalled. It is > slim and fairly fast, with embedded graphical debugger, class browser, > file browser... If you are into graphical IDEs you are probably going > to enjoy it. Windows only unfortunately. > > http://mmm-experts.com/Products.aspx?ProductId=4 Not to rain on anyone's parade, but I'd recommend using an IDE that isn't based on an obviously dead-end platform. (PyScripter is written in Delphi.) -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: Pros/Cons of Turbogears/Rails?
Fredrik Lundh wrote: > [EMAIL PROTECTED] wrote: > > P.S. If I wanted to provide an image by streaming the > > file data directly over the connection, rather than by > > referring to an image file, how would I do that? I'd > > like to build code that would allow images to be assembled > > into a single-file photo album (zip or bsddb file), and > > so can't refer to them as individual image files. > > some browsers support special data URL:s for images, but that doesn't > work well for large images, and isn't portable. on the other hand, I > don't really see why there has to be 1:1 mapping between URL:s and image > files on your machine, especially if you're using a web application > framework... I think he's just asking "how do you specify a mimetype and stream binary data from within the framework." IIANM in TurboGears all you have to do is return a file-like object from your page handler. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: postgresql plpython bug
Mage wrote: > create or replace function trigger_keywords_maintain() returns trigger as $$ > return 'MODIFY' > $$ language plpythonu; > > update table set id = id where id = 7; > > ERROR: invalid input syntax for type timestamp: "2005-05-03 > 14:07:33,279213" > > I see that Python's timestamp format is not accepted by postgresql. First, you don't give enough context to see where your python code generates a timestamp, but in any case it's more of a limitation than a bug that plpython doesn't try to autoconvert certain datatypes. (Are you even returning a datetime class, or a string?) You could play around with strftime to try to get something postgresql will recognize, but it's probably easier to just return an epoch value which you can turn into a postgresql timestamp with the abstime function. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: Capture close window button in Tkinter
William Gill wrote:
> I am trying to make a simple data editor in Tkinter where each data
> element has a corresponding Entry widget. I have tried to use the
> FocusIn/FocusOut events to set a 'hasChanged' flag (if a record has not
> changed, the db doesn't need updating). This seems to work fine except
> that when the user finishes and clicks a 'done' button or the close
> window button (in the root widget) no FocusOut event is triggered. I
> can trigger a FocusOut event if the 'done' button opens another window
> (i.e. a messagebox) that takes focus. Enter and Leave follow the mouse,
> but don't trigger when the user tabs between fields.
>
> Is there a better way to monitor 'hasChanged'?
I'd go with monitoring keypresses in the Entry widget.
> Also, how do I capture
> the root window close button?
root = Tk()
root.protocol("WM_DELETE_WINDOW", onexit)
--
http://mail.python.org/mailman/listinfo/python-list
Re: A critic of Guido's blog on Python's lambda
Randal L. Schwartz wrote: > > "Alex" == Alex Martelli <[EMAIL PROTECTED]> writes: > > Alex> The difference, if any, is that gurus of Java, C++ and Python get to > Alex> practice and/or keep developing their respectively favorite languages > Alex> (since those three are the "blessed" general purpose languages for > Alex> Google - I say "general purpose" to avoid listing javascript for > Alex> within-browser interactivity, SQL for databases, XML for data > Alex> interchange, HTML for web output, &c, &c), while the gurus of Lisp, > Alex> Limbo, Dylan and Smalltalk don't (Rob Pike, for example, is one of the > Alex> architects of sawzall -- I already pointed to the whitepaper on that > Alex> special-purpose language, and he co-authored that paper, too). > > That's crazy. Some of the key developers of Smalltalk continue to work > on the Squeak project (Alan Kay, Dan Ingalls, and I'm leaving someone > out, I know it...). So please remove Smalltalk from that list. I thought it was clear that Alex was talking about "smalltalk gurus who work for Google." -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: Pycon disappointment
On Mar 16, 10:20 am, Barry Hawkins <[EMAIL PROTECTED]> wrote: > I shared the same perception as Bruce; most "keynotes" > and lightning talks were anemic vendor pitches that really gutted the > spirit of what I experienced last year. I don't think you can lump the keynotes in with the lightning talks. I had to go check the schedule to see which keynotes were "diamond" ones. I wasn't thinking to myself, "oh, this must be a paid keynote" at the time at all. In fact, the Google one was the most entertaining of all, judging by audience reaction. But the vast majority of the vendor lightning talks were a waste of time, I agree. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: whats your favourite object relational mapper?
Serge Orlov wrote: > Flavio wrote: > > With so many object relational mappers out there, I wonder which one is > > the preferred tool among the Pythonists... is there a favourite? > > > > Sqlobject, PyDO, SQLAlchemy, dejavu, etc... > > Google results: > Sqlobject ORM: about 17,100 > PyDO ORM: 469 > SQLAlchemy ORM: 571 > dejavu ORM: 659 ... which, of course, goes to show how stupid a metric this is, now that even Ian Bicking has admitted that SqlObject in its current form is a dead end. Personally, I think SqlAlchemy has the brightest future. It's significantly more sophisticated than the others, and it's already quite usable and even stable (if the 0.1.3 to 0.1.4 transition is any indication), although I think technically still alpha. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: whats your favourite object relational mapper?
Giovanni Bajo wrote: > Jonathan Ellis wrote: > > > ... which, of course, goes to show how stupid a metric this is, now > > that even Ian Bicking has admitted that SqlObject in its current form > > is a dead end. > > > Got a pointer? http://blog.ianbicking.org/sqlobject-2.html -- http://mail.python.org/mailman/listinfo/python-list
Re: whats your favourite object relational mapper?
Steve Holden wrote: > I think describing this as Ian saying the code in its current form "is a > dead end" is to read rather more into the words than is actually there. Well, that may be. However, given that the 0.x code is so crufty that the v2 "refactor" is a multi-day (-week, now) process that merits a new project name, and there are enough architecture warts that it's not worth it to keep v2 backwards compatible, I'm not sure what requirements of being a dead end are missing here. :) I suppose that in one sense no OSS project is a dead end since you can always pick up the pieces yourself, but it's clear the 0.x series is not a place to expect much in the way of new developments from its author. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: Threads and sys.excepthook
Jesus Rivero - (Neurogeek) wrote: > Original exception was: > Unhandled exception in thread started by > Error in sys.excepthook: > > Original exception was: > > And if put a time.sleep(1) after the thread.start_new(test, > (name,)) #(1) part, then it does it all perfectly. Looks like the interpreter is shutting down before all the exception processing finishes. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: MOO meets Python
Aahz wrote: > http://playsh.org/ > http://sourceforge.net/projects/playsh Damn, I thought you meant MOO as in Master of Orion. -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
Re: Why aren't these more popular (Myghty & Pylons)?
Karlo Lozovina wrote: > There's only one thing bothering me, and that is it's lack of publicity. > There are only few posts on thig NG with some refference to Myghty, and > even less when it comes to Pylons (http://pylonshq.com/), framework built > on top of Myghy. Myghy project isn't that new to explain why people don't > know about it, so is something fundamentaly wrong with it? The whims of mindshare are indeed mysterious. > Anyway, to make a long story short, those who haven't heard about it, be > sure to check it out, you just might like it, but I would really like to > hear from people (if there are such) using it in real world > applications... http://techspot.zzzeek.org/index.php?/archives/7-Bittorrent.com,-running-Myghty-Again.html -Jonathan -- http://mail.python.org/mailman/listinfo/python-list
