Re: desperately looking for a howto on running my wxPython app on Vista
Mike Driscoll wrote: Hmmm...I'm not familiar with that DLL, but a little googling seems to indicate that you may be able to get it off your installation CD: it actually is there on my system. So this may be the red herring the Dependency Walker FAQ is warning for maybe I should leave this mess and revert to python 2.5 http://www.techimo.com/forum/applications-operating-systems/76550-winxp-msgina-dll-shlwapi-dll-problem.html The link above gives the steps for XP, but Vista will probably be similar. I don't know for sure, but you may have to register the dll once you've got it copied back onto your machine. Mike -- http://mail.python.org/mailman/listinfo/python-list
Re: desperately looking for a howto on running my wxPython app on Vista
Mike Driscoll wrote: On Apr 29, 4:17 am, Paul Sijben paul.sij...@xs4all.nl wrote: Is there any way to check which is the offending pyd/dll? (normally Vista does not give out much data on what went wrong) Paul You might be able to find it using the Dependency Walker utility: http://www.dependencywalker.com/ Mike, thanks! That spotted indeed some issues with one specific file: SHLWAPI.DLL, now let's see what is the issue. The report from Dependency Walker:Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module. is not telling me much at this moment. The FAQ from Dependency Walker sais this is not a major issue, but apparently on Vista it is ?!? Does anyone have a clue what I am to do about this? Interestingly On WinXP it flags DWMAPI.DLL as missing. I did not spot that on the Vista box. Paul - Mike -- http://mail.python.org/mailman/listinfo/python-list
Re: desperately looking for a howto on running my wxPython app on Vista
Gabriel Genellina wrote: I am currently stuck on the infamous R6034 error but I understand that after that there may be another issue with certain wxPython functions. That should be fixed in Python 2.6.2, I think. Are you compiling all your dependencies, including Python itself? R6034 is likely a DLL mismatch between parts of your project. I am using 2.6.2 and am compiling only my own changed modules. For all the other support modules I have taken the most recent ones (win32, wxpython) Is there any way to check which is the offending pyd/dll? (normally Vista does not give out much data on what went wrong) Paul -- http://mail.python.org/mailman/listinfo/python-list
desperately looking for a howto on running my wxPython app on Vista
python 2.6, py2exe and Vista do not make a happy set. Unfortunately I am in dire need to launch my app not only on WinXP but also on Vista. I need 2.6 because of a number of support packages I am using and some of which I am compiling myself (and python 2.5 needs a version of visual studio that is no longer available.) I can find all kinds of advice and discussions with google but I have yet to find a clear explanation on what a poor developer like me is supposed to do to get my python 2.6.2 , stackless, wxpython and all my (self-built) pyds to work happily on Vista. I am currently stuck on the infamous R6034 error but I understand that after that there may be another issue with certain wxPython functions. Can someone please point me to a howto on crafting the right Setup.py and manifests etc. to make this work? Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: default shelve on linux corrupts, does different DB system help?
Thanks very much for a clear and concise explanation of the problem and the solution! I am implementing it now in my system. Luckily we caught this one during testing so no important data has been lost. Unfortunately windows does not seem to support gdbm. But in our case, everything that is on the windows client is also available on the linux server, so we can recreate the DB at the expense of some bandwidth in case of failures. Paul s...@pobox.com wrote: Paul I have the problem that my shelve(s) sometimes corrupt (looks like Paul it has after python has run out of threads). Paul I am using the default shelve so on linux I get the dbhash Paul version. Is there a different DB type I can choose that is known Paul to be more resilient? And if so, what is the elegant way of doing Paul that? You don't say what version of Python you're using or what version of the Berkeley DB library underpins your installation, but I am going to guess it is 1.85. This has been known to have serious bugs for over a decade. (Just in the hash file implementation. The btree and recnum formats are ok. Unfortunately, the hash file implementation is what everybody has always gravitated to. Sort of like moths to a flame...) If that's the case, simply pick some other dbm file format for your shelves, e.g.: import gdbm import shelve f = gdbm.open(/tmp/trash.db, c) f.close() db = shelve.open(/tmp/trash.db) db[mike] = sharon db[4] = 5 db.keys() ['4', 'mike'] db.close() f = gdbm.open(/tmp/trash.db, c) f.keys() ['4', 'mike'] f['4'] 'I5\n.' f['mike'] S'sharon'\np1\n. As for uncorrupting your existing database, see if your Linux distribution has a db_recover program. If it does, you might be able to retrieve your data, though in the case of BerkDB 1.85's hash file I'm skeptical that can be done. I hope you weren't storing something valuable in it like your bank account passwords. -- http://mail.python.org/mailman/listinfo/python-list
default shelve on linux corrupts, does different DB system help?
I have the problem that my shelve(s) sometimes corrupt (looks like it has after python has run out of threads). I am using the default shelve so on linux I get the dbhash version. Is there a different DB type I can choose that is known to be more resilient? And if so, what is the elegant way of doing that? Paul -- http://mail.python.org/mailman/listinfo/python-list
import * and py2exe
I am trying to turn my application into a WinXP exe. Py2exe has packaged all my files up into one humongous executable. When trying to run the app, it complains that it can not find modules I just saw it include. These invariably are modules that have been imported using from modulename import * Apparently this confuses py2exe. Well I tried unconfusing it by giving those modules as imports in the first place! How can I convince the py2exe-generated app to look for those modules in its own .exe file??? Paul -- http://mail.python.org/mailman/listinfo/python-list
giving imp.load_module not a real file, HOW?
I am running into a problem with the python interpreter's internals. For some reason imp.load_module insists on getting a real open file as the second parameter. I have not able to fool it with stringIO or overloaded file objects. So now I have two questions: 1) why does load_module insist on a real file 2) is there a way around it that does not involve (say) tempfile? Paul -- http://mail.python.org/mailman/listinfo/python-list
Pyrex bitten by the exception bug
I am running into the (apparently) well-known issue with pyrex that trying to raise an exception using python2.5 and pyrex will lead to a TypeError, as TypeError: exceptions must be strings, classes, or instances, not exceptions.ImportError Is there a fixc for this issue? best regards, Paul Sijben -- http://mail.python.org/mailman/listinfo/python-list
Re: Pyrex bitten by the exception bug
Stefan Behnel wrote: Paul Sijben wrote: I am running into the (apparently) well-known issue with pyrex that trying to raise an exception using python2.5 and pyrex will lead to a TypeError, as TypeError: exceptions must be strings, classes, or instances, not exceptions.ImportError You should use a recent Pyrex version. I have :-( the most recent from their site But using Cython is a better idea anyway. I will look into that. Thanks, Paul Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: encrypting python modules
How often do these things *actually* happen? Of those that actually do it, how many are clueless enough that when they run into problems they blame you for it? (And remember that you won't even find out about the non-clueless ones.) This is a rethorical question, right? -- http://mail.python.org/mailman/listinfo/python-list
Re: encrypting python modules
Mike, thanks for the constructive feedback.Indeed i probably need to patch import in some way. Looks like there is no standard way to get this done. So I guess I have do it myself... In the famous last words department: how hard can that be? ;-) Paul Mike Meyer wrote: On Sat, 12 Jan 2008 09:47:26 +1100 Ben Finney [EMAIL PROTECTED] wrote: Paul Sijben [EMAIL PROTECTED] writes: I know that I can not stop a dedicated hacker deconstructing my code. A direct consequence of this is that you can not stop *anyone* from deconstructing your code if it's in their possession. It takes only one dedicated, skilled person to crack your obfuscation system and distribute an automated process for doing so to anyone interested. Except that's not what he's trying to do. However I can not imagine that I would be the first one planning to do this. So is there a solution like this available somewhere? Trying to make bits uncopyable and unmodifiable is like trying to make water not wet. And again, that's not what he's trying to do. He wants to arrange things so that he doesn't have to support unmodified versions of his code, by making it impossible to import modified modules. While that's still impossible, once you decide how difficult you want to make it for people to do that, you can *probably* make it that difficult - but the process gets progressively more difficult and expensive as you make it harder. I think he's contemplating only the simplest, least expensive step: adding an import hook that only allows imports of digitally signed modules. If planning to deploy on Windows, where he has to bundle a python with his application, he may well implement the hook in the interpreter instead of in python, so it's harder to find. If you wanted to go to the expense, you could probably arrange things so that the digital signatures are the more vulnerable attack vectors, but I'd expect to spend millions of dollars doing so. mike -- http://mail.python.org/mailman/listinfo/python-list
Re: encrypting python modules
Robert Latest wrote: Paul Sijben wrote: The problem: I have a client-server app written in python. I want to make sure that the client is not: 1) destabilized by users accidentally or on purpose dropping python files in the path (after which calling the helpdesk will not be useful) 2) extended with new features without me knowing about it (again resulting in calls to my helpdesk...) You could check the MD5 hashes of your files. robert indeed but I still need to hook into import to do that reliably, right? -- http://mail.python.org/mailman/listinfo/python-list
encrypting python modules
Hello, this question has come by repeatedly in several guises over the past years but has never been solved in this forum as far as I have been able to Google. However since so many people are asking the question, I hope someone has made a solution and is willing to share it. The problem: I have a client-server app written in python. I want to make sure that the client is not: 1) destabilized by users accidentally or on purpose dropping python files in the path (after which calling the helpdesk will not be useful) 2) extended with new features without me knowing about it (again resulting in calls to my helpdesk...) 3) trivially decompiled. Added issue, I want the client to be able to update itself when I have fixed bugs or created new functionality. I know that I can not stop a dedicated hacker deconstructing my code. Since all clients need to go through the server I am not afraid of freeloaders. I am now considering overriding import with some code that will only import modules signed and crypted by me. However I can not imagine that I would be the first one planning to do this. So is there a solution like this available somewhere? Paul Sijben -- http://mail.python.org/mailman/listinfo/python-list
Re: finding bluetooth serial port
thanks very much! Grant Edwards wrote: On 2007-11-07, Paul Sijben [EMAIL PROTECTED] wrote: To automate/ease configuration in my app I am trying to find out to which serial port a certain bluetooth device is connected. With pybluez I can find out which bluetooth devices I have, but it will not tell me the serial port they are mapped to. Is there a way to figure this out from python? (I am insterested in the platforms WinXP and linux primarily) Under linux, the right thing to do is to write a udev rule so that the device has a predictiable name (or symlink). http://reactivated.net/writing_udev_rules.html If you don't want to write a udev rule, you'll need to bascally re-implement udev by parsing the sysfs directory tree until you find the device you're looking for. Here's how to do it for USB (I assume BT works in a similar fashion). Let's say I know the device has vendor ID 0403, product ID 6001, and serial number 123456. I search through the directories under /sys/devices until I find a directory containing three files named idProduct idVendor serial Which contain the three strings I'm looking for. In this case: # cat /sys/devices/pci:00/:00:10.3/usb5/5-1/idVendor 0403 # cat /sys/devices/pci:00/:00:10.3/usb5/5-1/idProduct 6001 # cat /sys/devices/pci:00/:00:10.3/usb5/5-1/serial 12345678 Once you've found that directory, you can look at the other entries to find out whatever you want to know about the device: /sys/devices/pci:00/:00:10.3/usb5/5-1/ |-- 5-1:1.0 | |-- bAlternateSetting | |-- bInterfaceClass | |-- bInterfaceNumber | |-- bInterfaceProtocol | |-- bInterfaceSubClass | |-- bNumEndpoints | |-- bus - ../../../../../../bus/usb | |-- driver - ../../../../../../bus/usb/drivers/ftdi_sio | |-- ep_02 - ../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0/usbdev5.42_ep02 | |-- ep_81 - ../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0/usbdev5.42_ep81 | |-- interface | |-- modalias | |-- power | | |-- state | | `-- wakeup | |-- subsystem - ../../../../../../bus/usb | |-- ttyUSB0 | | |-- bus - ../../../../../../../bus/usb-serial | | |-- driver - ../../../../../../../bus/usb-serial/drivers/ftdi_sio | | |-- power | | | |-- state | | | `-- wakeup | | |-- subsystem - ../../../../../../../bus/usb-serial | | |-- tty:ttyUSB0 - ../../../../../../../class/tty/ttyUSB0 | | `-- uevent | |-- uevent | |-- usb_endpoint:usbdev5.42_ep02 - ../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0/usbdev5.42_ep02 | |-- usb_endpoint:usbdev5.42_ep81 - ../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0/usbdev5.42_ep81 | |-- usbdev5.42_ep02 | | |-- bEndpointAddress | | |-- bInterval | | |-- bLength | | |-- bmAttributes | | |-- dev | | |-- device - ../../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0 | | |-- direction | | |-- interval | | |-- power | | | |-- state | | | `-- wakeup | | |-- subsystem - ../../../../../../../class/usb_endpoint | | |-- type | | |-- uevent | | `-- wMaxPacketSize | `-- usbdev5.42_ep81 | |-- bEndpointAddress | |-- bInterval | |-- bLength | |-- bmAttributes | |-- dev | |-- device - ../../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0 | |-- direction | |-- interval | |-- power | | |-- state | | `-- wakeup | |-- subsystem - ../../../../../../../class/usb_endpoint | |-- type | |-- uevent | `-- wMaxPacketSize |-- bConfigurationValue |-- bDeviceClass |-- bDeviceProtocol |-- bDeviceSubClass |-- bMaxPacketSize0 |-- bMaxPower |-- bNumConfigurations |-- bNumInterfaces |-- bcdDevice |-- bmAttributes |-- bus - ../../../../../bus/usb |-- configuration |-- devnum |-- driver - ../../../../../bus/usb/drivers/usb |-- ep_00 - ../../../../../devices/pci:00/:00:10.3/usb5/5-1/usbdev5.42_ep00 |-- event_char |-- idProduct |-- idVendor |-- manufacturer |-- maxchild |-- power | |-- state | `-- wakeup |-- product |-- serial |-- speed |-- subsystem - ../../../../../bus/usb |-- uevent |-- usb_device:usbdev5.42 - ../../../../../class/usb_device/usbdev5.42 |-- usb_endpoint:usbdev5.42_ep00 - ../../../../../devices/pci:00/:00:10.3/usb5/5-1/usbdev5.42_ep00 |-- usbdev5.42_ep00 | |-- bEndpointAddress | |-- bInterval | |-- bLength | |-- bmAttributes | |-- dev | |-- device - ../../../../../../devices/pci:00/:00:10.3/usb5/5-1 | |-- direction | |-- interval | |-- power | | |-- state | | `-- wakeup | |-- subsystem - ../../../../../../class/usb_endpoint | |-- type | |-- uevent | `-- wMaxPacketSize `-- version If you want to search
finding bluetooth serial port
To automate/ease configuration in my app I am trying to find out to which serial port a certain bluetooth device is connected. With pybluez I can find out which bluetooth devices I have, but it will not tell me the serial port they are mapped to. Is there a way to figure this out from python? (I am insterested in the platforms WinXP and linux primarily) Paul Sijben -- http://mail.python.org/mailman/listinfo/python-list
Re: python 2.5.1 segfault, multithreading dual core issue?
thanks very much! I am currently compiling python with the patch and will test it over the coming days. Paul Hrvoje Niksic wrote: Paul Sijben [EMAIL PROTECTED] writes: I am running a multi-threaded python application in a dual core intel running Ubuntu. [...] Judging from the stack trace, this patch has a good chance of fixing your problem: http://mail.python.org/pipermail/python-dev/2007-August/074232.html -- http://mail.python.org/mailman/listinfo/python-list
python 2.5.1 segfault, multithreading dual core issue?
I am running a multi-threaded python application in a dual core intel running Ubuntu. I am using python 2.5.1 that I compiled myself. At random points I am getting segmentation faults (sometimes indicating a duplicate free). Below is the backtrace of the latest segfault. I am thinking this might be an issue related to the dual core CPU so I am now running the app with affinity to one CPU to test this hypothesis. Where can I put such a bugreport? Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1486967920 (LWP 4579)] 0x009196ed in fclose@@GLIBC_2.1 () from /lib/libc.so.6 (gdb) bt #0 0x009196ed in fclose@@GLIBC_2.1 () from /lib/libc.so.6 #1 0x0806af9b in file_close (f=0xa259e30) at Objects/fileobject.c:446 #2 0x080c51b0 in PyEval_EvalFrameEx (f=0xa3c790c, throwflag=0) at Python/ceval.c:3548 #3 0x080c5795 in PyEval_EvalFrameEx (f=0xa3c77a4, throwflag=0) at Python/ceval.c:3650 #4 0x080c5795 in PyEval_EvalFrameEx (f=0xa3afccc, throwflag=0) at Python/ceval.c:3650 #5 0x080c5795 in PyEval_EvalFrameEx (f=0xa3c8804, throwflag=0) at Python/ceval.c:3650 #6 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f315c0, globals=0xb7f29824, locals=0x0, args=0xa39b9a8, argcount=3, kws=0xa39b9b4, kwcount=0, defs=0xb7c8fb98, defcount=1, closure=0x0) at Python/ceval.c:2831 #7 0x080c4a59 in PyEval_EvalFrameEx (f=0xa39b864, throwflag=0) at Python/ceval.c:3660 #8 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f2ec38, globals=0xb7f29824, locals=0x0, args=0xa399920, argcount=3, kws=0xa39992c, kwcount=0, defs=0xb7c8fad8, defcount=1, closure=0x0) at Python/ceval.c:2831 #9 0x080c4a59 in PyEval_EvalFrameEx (f=0xa3997c4, throwflag=0) at Python/ceval.c:3660 #10 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f2eda0, globals=0xb7f29824, locals=0x0, args=0xa340e30, argcount=3, kws=0xa340e3c, kwcount=0, defs=0xb7cea330, defcount=4, closure=0x0) at Python/ceval.c:2831 #11 0x080c4a59 in PyEval_EvalFrameEx (f=0xa340cb4, throwflag=0) at Python/ceval.c:3660 #12 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f317b8, globals=0xb7f29824, locals=0x0, args=0xa39cedc, argcount=2, kws=0xa39cee4, kwcount=2, defs=0xb7c905b0, defcount=3, closure=0x0) at Python/ceval.c:2831 #13 0x080c4a59 in PyEval_EvalFrameEx (f=0xa39cd7c, throwflag=0) at Python/ceval.c:3660 #14 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f316e0, globals=0xb7f29824, locals=0x0, args=0xa3aff8c, argcount=4, kws=0xa3aff9c, kwcount=1, defs=0xb7c8fbb8, defcount=2, closure=0x0) at Python/ceval.c:2831 #15 0x080c4a59 in PyEval_EvalFrameEx (f=0xa3afe34, throwflag=0) at Python/ceval.c:3660 #16 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f07728, globals=0xb7f59acc, locals=0x0, args=0xa32b9ac, argcount=3, kws=0xa32b9b8, kwcount=0, defs=0xb7c0a1f8, defcount=2, closure=0x0) at Python/ceval.c:2831 #17 0x080c4a59 in PyEval_EvalFrameEx (f=0xa32b86c, throwflag=0) at Python/ceval.c:3660 #18 0x080c5795 in PyEval_EvalFrameEx (f=0xa3c84f4, throwflag=0) at Python/ceval.c:3650 #19 0x080c5795 in PyEval_EvalFrameEx (f=0xa3a4ef4, throwflag=0) at Python/ceval.c:3650 #20 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f1d1d0, globals=0xb7f11dfc, locals=0x0, args=0xa2d8cd8, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2831 #21 0x0810d6a1 in function_call (func=0xb7f26e2c, arg=0xa2d8ccc, kw=0x0) at Objects/funcobject.c:517 #22 0x0805a257 in PyObject_Call (func=0xa48ff4, arg=0xa2d8ccc, kw=0x0) at Objects/abstract.c:1860 #23 0x08060387 in instancemethod_call (func=0xa23b2fc, arg=0xa2d8ccc, kw=0x0) at Objects/classobject.c:2497 #24 0x0805a257 in PyObject_Call (func=0xa48ff4, arg=0xb7f4102c, kw=0x0) at Objects/abstract.c:1860 #25 0x080be79c in PyEval_CallObjectWithKeywords (func=0xa23b2fc, arg=0xb7f4102c, kw=0x0) at Python/ceval.c:3433 #26 0x080f01b8 in t_bootstrap (boot_raw=0xa3c8048) at ./Modules/threadmodule.c:424 #27 0x00a3a45b in start_thread () from /lib/libpthread.so.0 #28 0x0099223e in clone () from /lib/libc.so.6 -- http://mail.python.org/mailman/listinfo/python-list
Re: Windows XP timezone language issue
MRAB wrote:
On Jun 13, 7:31 am, Paul Sijben [EMAIL PROTECTED] wrote:
I ran into an internationalization issue. I need a consistent idea about
the timezone my application is running on. However when I run the following:
import time
time.tzname
I get back ('West-Europa (standaardtijd)', 'West-Europa (zomertijd)')
which is in dutch (the language of the host machine) and verbose.
I wanted to get ('CEST','CET') or something international so I can work
with itin the same way on all platforms.
That is the right way to find out the timezone in a consistent way
across platforms (windows/linux/mac) and languages?
Well, time.timezone will return the timezone as an integer.
true, I am wondering if that is enough for my app. (with some of the odd
timezones in the world and DTS issues and all
--
http://mail.python.org/mailman/listinfo/python-list
Windows XP timezone language issue
I ran into an internationalization issue. I need a consistent idea about
the timezone my application is running on. However when I run the following:
import time
time.tzname
I get back ('West-Europa (standaardtijd)', 'West-Europa (zomertijd)')
which is in dutch (the language of the host machine) and verbose.
I wanted to get ('CEST','CET') or something international so I can work
with itin the same way on all platforms.
That is the right way to find out the timezone in a consistent way
across platforms (windows/linux/mac) and languages?
Paul
--
http://mail.python.org/mailman/listinfo/python-list
Re: audio video streaming communications
Hi Ken, I am looking for something similar. I can do the communications myself but need to be able to select a video feed, capture it and also need to display it through wxPython. Trawled the web and even tried to hire coders to create it for me. So far I have been having no luck. I did learn that fastaudio is a good way to bluescreen a WinXP box ;-) Have you found something? Paul Ken Seehart wrote: Hello, I am looking for a good audio/video conferencing library. Ideally it should work with wxPython (or have some means of making it work there). So far my main difficulty in my attempt at searching for such a package is that there is so much stuff out there on downloading music and videos. I am not interested in download torrents, etc. I'm just looking video conferencing, and live video broadcasts, etc. Any recommendations? Thanks, - Ken -- http://mail.python.org/mailman/listinfo/python-list
worker thread catching exceptions and putting them in queue
All, in a worker thread setup that communicates via queues is it possible to catch exceptions raised by the worker executed, put them in an object and send them over the queue to another thread where the exception is raised in that scope? considering that an exception is an object I feel it ought to be possible, however I do not see how to go about it. does anyone have a pointer towards the solution? Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: worker thread catching exceptions and putting them in queue
Stargaming Diez, thanks very much! Stargaming wrote: Paul Sijben schrieb: All, in a worker thread setup that communicates via queues is it possible to catch exceptions raised by the worker executed, put them in an object and send them over the queue to another thread where the exception is raised in that scope? considering that an exception is an object I feel it ought to be possible, however I do not see how to go about it. does anyone have a pointer towards the solution? Paul You're right, even exceptions are objects in Python. For further studies, read http://docs.python.org/lib/module-exceptions.html You can catch an exception like this: try: worker.do_some_work_that_may_raise_an_exception() except Exception, e: # the first argument is the type of error you want to handle # it is Exception here, the baseclass of all computation exceptions # the second argument is the variable (name) where you want to save # the specific exception raised to # it's 'e' here, a common shortcut for exception exception_handler.handle(e) # notice that you can pass e around as you like For further information on exceptions and how to handle them, read chapter 8 of the tutorial, especially starting from 8.3: http://docs.python.org/tut/node10.html#SECTION001030 P.S. I don't know if what I told still applies to Python 3.0 -- a lot of changes are upcoming related to exception raising and handling. -- http://mail.python.org/mailman/listinfo/python-list
Re: maximum number of threads
Gabriel Genellina wrote: Simply you can't, as you can't have 1 open files at once. Computer resources are not infinite. sure but *how* fast they run out is the issue here Do you really need so many threads? I might be able to do with a few less but I still need many. I have done a quick test. on WinXP I can create 1030 threads on Fedora Core 6 I can only create 301 (both python2.4 and 2.5) now the 301 is rather low I'd say. Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: maximum number of threads
Felipe Almeida Lessa wrote: Maybe Stackless could help the OP? http://www.stackless.com/ thanks I will look into it! -- http://mail.python.org/mailman/listinfo/python-list
Re: convert binary data to int
in some cases struct.unpack would also help [EMAIL PROTECTED] wrote: Hello, I need to convert a 3 byte binary string like \x41\x00\x00 to 3 int values ( (65,0,0) in this case). The string might contain characters not escaped with a \x, like A\x00\x00 Any ideas? Daniel -- http://mail.python.org/mailman/listinfo/python-list
Re: maximum number of threads
All thanks for all the input! This was very informative. Looks like I indeed need stackless as my code benefits from being concurrently designed. Paul Jean-Paul Calderone wrote: On Wed, 10 Jan 2007 12:11:59 -0200, Felipe Almeida Lessa [EMAIL PROTECTED] wrote: On 1/10/07, Laurent Pointal [EMAIL PROTECTED] wrote: This is a system configurable limit (up to a maximum). See ulimit man pages. test ulimit -a to see what are the current limits, and try with ulimit -u 2000 to modify the maximum number of user process (AFAIK each thread use a process entry on Linux) I don't think it's only this. Indeed you are correct. The actual limit you are hitting is the size of your address space. Each thread is allocated 8MB of stack. 382 threads consumes about 3GB of address space. Even though most of this memory isn't actually allocated, the address space is still used up. So, when you try to create the 383rd thread, the kernel can't find anyplace to put its stack. So you can't create it. Try reducing your stack size or reducing the number of threads you create. There's really actually almost no good reason to have this many threads, even though it's possible. [EMAIL PROTECTED]:~$ python Desktop/test.py 50 100 150 200 250 300 350 Exception raised: can't start new thread Biggest number of threads: 382 [EMAIL PROTECTED]:~$ ulimit -Ss 4096 [EMAIL PROTECTED]:~$ python Desktop/test.py 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 Exception raised: can't start new thread Biggest number of threads: 764 [EMAIL PROTECTED]:~$ Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
maximum number of threads
I have a server in Python 2.5 that generates a lot of threads. It is running on a linux server (Fedora Core 6). The server quickly runs out of threads. I am seeing the following error. File /home/sijben/ORCA/src/libmercury_mt.py, line 565, in __init__ MercuryObject.__init__(self,mylink) File /home/sijben/ORCA/src/libmercury_mt.py, line 223, in __init__ self.start() File /usr/local/lib/python2.5/threading.py, line 434, in start _start_new_thread(self.__bootstrap, ()) error: can't start new thread Does anyone know what it going on here and how I can ensure that I have all the threads I need? Paul -- http://mail.python.org/mailman/listinfo/python-list
Python UPnP on Linux?
I am writing some client software that needs to accept connections from the Internet. So I am looking for a UPnP implementation for a client of an Internet gateway so I can request the NAT binding. Googling on this I have found win32 implementations and Twisted implementations yet I am looking for a way to do it on Linux WITHOUT Twisted. Who knows a pointer? Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Python UPnP on Linux?
You are right of course but I was hoping to avoid that. Twisted is very large and has all kinds of internal dependencies. Sybren Stuvel wrote: Paul Sijben enlightened us with: Googling on this I have found win32 implementations and Twisted implementations yet I am looking for a way to do it on Linux WITHOUT Twisted. Twisted is Open Source, so you could browse the source and see how they do it. Sybren -- http://mail.python.org/mailman/listinfo/python-list
Re: FOUNDIT (was Re: massive threading performance)
Lawrence D'Oliveiro wrote: In article [EMAIL PROTECTED], Paul Sijben [EMAIL PROTECTED] wrote: I found that the problem was caused by the sending thread not giving control back quickly enough to the receiving thread. Also in going through the code I found an old self.s.setblocking(0)call that was no longer relevant. Removing that solved my problem. Something that took 20 seconds now takes just 1. You might also find that it goes still faster if you forego threading and use a select.select loop. thanks for that. I will have to try it. -- http://mail.python.org/mailman/listinfo/python-list
UDP performance
I am stumped by the following problem. I have a large multi-threaded server accepting communications on one UDP port (chosen for its supposed speed). I have been profiling the code and found that the UDP communication is my biggest drain on performance! Communication where the client and the server are on the same machine still takes 300ms or sometimes much more per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64). I must be doing something wrong and would really appreciate feedback on my code below: I open the server port with self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.s.bind((myaddress, myport)) I then open a client port with self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM) self.s.connect((host, port)) the client sends data with self.s.sendall(data) and the server with self.s.sendto(data,link.remoteaddress) both receive with buf, address = socket.recvfrom(8192) The sender and receiver are in separate threads (threading.Thread). Does anyone know what is going wrong here, the socket communication, the thread scheduling? Paul Sijben -- http://mail.python.org/mailman/listinfo/python-list
Re: UDP performance
Serge Orlov wrote: Paul Sijben wrote: I am stumped by the following problem. I have a large multi-threaded server accepting communications on one UDP port (chosen for its supposed speed). I have been profiling the code and found that the UDP communication is my biggest drain on performance! Communication where the client and the server are on the same machine still takes 300ms or sometimes much more per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64). [snip] buf, address = socket.recvfrom(8192) I'm not an expert here, but I AFAIK UDP packet size should be kept below 512 bytes otherwise it can cause fragmentation and hence all the slow stuff like acknoledgements, timeouts, etc... good point in general but in practice they are in this case. but this is less of a problem on the loopback interface anyway. -- http://mail.python.org/mailman/listinfo/python-list
Re: UDP performance
Serge Orlov wrote: Paul Sijben wrote: Serge Orlov wrote: Paul Sijben wrote: I am stumped by the following problem. I have a large multi-threaded server accepting communications on one UDP port (chosen for its supposed speed). I have been profiling the code and found that the UDP communication is my biggest drain on performance! Communication where the client and the server are on the same machine still takes 300ms or sometimes much more per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64). [snip] buf, address = socket.recvfrom(8192) I'm not an expert here, but I AFAIK UDP packet size should be kept below 512 bytes otherwise it can cause fragmentation and hence all the slow stuff like acknoledgements, timeouts, etc... good point in general but in practice they are in this case. but this is less of a problem on the loopback interface anyway. Isn't it still controlled by MTU even on the loopback? What is your MTU on the loopback? loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 eth0 Link encap:Ethernet HWaddr 00:14:85:35:A4:5D inet addr:192.168.0.124 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::214:85ff:fe35:a45d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 -- http://mail.python.org/mailman/listinfo/python-list
Re: UDP performance
Martin P. Hellwig wrote: [snip] Is the connection 1:1 i.e. the receiving end receives data only from one sender at the time? And how do you handle lost packages in your application? no the server is receiving from multiple clients. and at the moment I do not handle lost packets. -- http://mail.python.org/mailman/listinfo/python-list
FOUNDIT (was Re: massive threading performance)
I found that the problem was caused by the sending thread not giving
control back quickly enough to the receiving thread.
Also in going through the code I found an old self.s.setblocking(0)call
that was no longer relevant. Removing that solved my problem.
Something that took 20 seconds now takes just 1.
Thanks to those who took the time to respond to my earlier messages.
Paul
Paul Sijben wrote:
OK the problem I posted about earlier is NOT a UDP/socket problem, it is
a threading problem. Albeit one that only happens when you have many
thrreads
I have made two little scripts using the code I copied below, one client
and one server (attached).
If I run them concurrently on the same machine I see the following times:
1145526192.825508
1145526192.825848
1145526193.829325
1145526193.834927
a transfer time in the milliseconds. Much less than the times I see in
the full application.
OK so I put them both in a multithreaded script (also attached)
1145526971.619558
1145526971.619909
1145526972.619241
1145526972.619647
again transfer time in milliseconds.
Not like this that I get from the profile of my code:
1145517554.363850 send
1145517554.647485 recv
which uses the same communication and threading but has 20+ threads?
Now I am completely baffled!
again I really appreciate if anyone can shed light on this!
Paul
Paul Sijben wrote:
I am stumped by the following problem. I have a large multi-threaded
server accepting communications on one UDP port (chosen for its supposed
speed).
I have been profiling the code and found that the UDP communication is
my biggest drain on performance! Communication where the client and the
server are on the same machine still takes 300ms or sometimes much more
per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
I must be doing something wrong and would really appreciate feedback on
my code below:
I open the server port with
self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.s.bind((myaddress, myport))
I then open a client port with
self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.connect((host, port))
the client sends data with
self.s.sendall(data)
and the server with
self.s.sendto(data,link.remoteaddress)
both receive with
buf, address = socket.recvfrom(8192)
The sender and receiver are in separate threads (threading.Thread).
Does anyone know what is going wrong here, the socket communication, the
thread scheduling?
Paul Sijben
import socket
import time
profile=[]
s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('127.0.0.1', 42420))
print %f%time.time()
s.sendall('testerdetest')
time.sleep(1)
print %f%time.time()
s.sendall('testerdetest')
print %f%time.time()
import socket
import time
profile=[]
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 42420))
buf, address =s.recvfrom(8192)
print %f%time.time()
print buf, address
buf, address =s.recvfrom(8192)
print %f%time.time()
print buf, address
import socket
import time
import threading
profile=[]
class server(threading.Thread):
def __init__(self):
self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.s.bind(('', 42420))
threading.Thread.__init__(self)
self.start()
def run(self):
buf, address =self.s.recvfrom(8192)
print %f%time.time()
print buf, address
buf, address =self.s.recvfrom(8192)
print %f%time.time()
print buf, address
class client(threading.Thread):
def __init__(self):
self.s =socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.connect(('127.0.0.1', 42420))
threading.Thread.__init__(self)
self.start()
def run(self):
print %f%time.time()
self.s.sendall('testerdetest')
time.sleep(1)
print %f%time.time()
self.s.sendall('testerdetest')
print %f%time.time()
serv=server()
cl=client()
--
http://mail.python.org/mailman/listinfo/python-list
massive threading performance (was:Re: UDP performance)
OK the problem I posted about earlier is NOT a UDP/socket problem, it is
a threading problem. Albeit one that only happens when you have many
thrreads
I have made two little scripts using the code I copied below, one client
and one server (attached).
If I run them concurrently on the same machine I see the following times:
1145526192.825508
1145526192.825848
1145526193.829325
1145526193.834927
a transfer time in the milliseconds. Much less than the times I see in
the full application.
OK so I put them both in a multithreaded script (also attached)
1145526971.619558
1145526971.619909
1145526972.619241
1145526972.619647
again transfer time in milliseconds.
Not like this that I get from the profile of my code:
1145517554.363850 send
1145517554.647485 recv
which uses the same communication and threading but has 20+ threads?
Now I am completely baffled!
again I really appreciate if anyone can shed light on this!
Paul
Paul Sijben wrote:
I am stumped by the following problem. I have a large multi-threaded
server accepting communications on one UDP port (chosen for its supposed
speed).
I have been profiling the code and found that the UDP communication is
my biggest drain on performance! Communication where the client and the
server are on the same machine still takes 300ms or sometimes much more
per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
I must be doing something wrong and would really appreciate feedback on
my code below:
I open the server port with
self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.s.bind((myaddress, myport))
I then open a client port with
self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.connect((host, port))
the client sends data with
self.s.sendall(data)
and the server with
self.s.sendto(data,link.remoteaddress)
both receive with
buf, address = socket.recvfrom(8192)
The sender and receiver are in separate threads (threading.Thread).
Does anyone know what is going wrong here, the socket communication, the
thread scheduling?
Paul Sijben
import socket
import time
profile=[]
s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('127.0.0.1', 42420))
print %f%time.time()
s.sendall('testerdetest')
time.sleep(1)
print %f%time.time()
s.sendall('testerdetest')
print %f%time.time()
import socket
import time
profile=[]
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 42420))
buf, address =s.recvfrom(8192)
print %f%time.time()
print buf, address
buf, address =s.recvfrom(8192)
print %f%time.time()
print buf, addressimport socket
import time
import threading
profile=[]
class server(threading.Thread):
def __init__(self):
self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.s.bind(('', 42420))
threading.Thread.__init__(self)
self.start()
def run(self):
buf, address =self.s.recvfrom(8192)
print %f%time.time()
print buf, address
buf, address =self.s.recvfrom(8192)
print %f%time.time()
print buf, address
class client(threading.Thread):
def __init__(self):
self.s =socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.connect(('127.0.0.1', 42420))
threading.Thread.__init__(self)
self.start()
def run(self):
print %f%time.time()
self.s.sendall('testerdetest')
time.sleep(1)
print %f%time.time()
self.s.sendall('testerdetest')
print %f%time.time()
serv=server()
cl=client()--
http://mail.python.org/mailman/listinfo/python-list
