Re: Loading C extension from memory
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/13/2010 06:14 AM, mk wrote: > I wonder if there is a way to load C extension from in-memory object, > not from the file on the disk? > > I'm asking bc I would like to download C extensions over network and > load them into Python interpreter (without storing the C extension in > file on the disk). You could download the C source over the network, compile and use that. It has the advantage of working on all platforms where you have a compiler so you wouldn't have to have the same extension compiled for Mac, Linux, Windows, 32 and 64 bit. If the local machine doesn't have a compiler you can even use libtcc or something newer. For example see this 3 year old page, as well as links at the bottom: http://www.cs.tut.fi/~ask/cinpy/ Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkv0z4UACgkQmOOfHg372QQ1UQCg2+ODiHMzi36//gG3gK6j4Uib 2u8AoMI2LXU7BFYa7Jn5b+vF42/EnNYU =RWO+ -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.X vs. 3.X - level of acceptance?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/27/2010 03:57 AM, Stephan Schulz wrote: > Is Python 3 sucessful enough to make a switch worthwhile now? The language/interpreter is just fine. The biggest problem is 3rd party modules. My own module (APSW) has been available since the early betas of Python 3 and I make it available for Python 2.3 onwards. I have a release every month or two, and so you can see adoption trends for Windows users over time: http://code.google.com/p/apsw/downloads/list?can=1&q=binary Python 2.6 is by far the most popular, but 3.1 is picking up. It is disturbing just many 2.3 users there still are though. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvXFTUACgkQmOOfHg372QR7YwCgpZdsgkV9iBy/1/8eIBwxy3S4 Sy8AoI8vsivExNADG9Bmx+WbWTQN74VD =0lxT -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 3.0 usage?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Philip Semanchuk wrote: > is Python 3.0 seeing use in production > anywhere, or did most of the Python world move to 3.1 as soon as it was > released? Python 3.0 has been end of lifed: http://www.python.org/download/releases/3.0.1/ Consequently no one should be using it in production. Incidentally my own extension is in a similar situation to yours - Python 2.3 onwards is supported for the 2.x family. I did support 3.0 from beta releases onwards but various changes in the library between 3.0 and 3.1 and the end of lifing of 3.0 meant I dropped 3.0 support. No one has asked for it since. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkt8zfoACgkQmOOfHg372QS2cACgznTcQXRcxMqqlfIt4F0C2TaX d9IAoNhkY1dvX1aKlONyVGIL2zplNNF7 =jM8O -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Py3: Terminal or browser output?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gnarlodious wrote: > I want to have a script > output HTML if run in a browser and plain text if run in a Terminal. You may also want to look into urwid. It provides you with a text console interface but can also provide HTML. It has widgets like text boxes, lists, tick boxes etc. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkt3JWUACgkQmOOfHg372QRifACfYKS7+bmt6F3WPXYatM7yKVcs knMAoLx5sJ3lFmofrrgHaS3aYOBAun0d =Nxd4 -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: execute sqlite3 dot commands in python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 gintare statkute wrote: > Does anybody know if it possible to execute sqlite3 dot commands in python? The dot commands are parsed and executed by different code not part of the standard SQLite library. However if you want interactive shell functionality from Python then you can use APSW. It includes a shell you can just go ahead and use based on a shell class you can extend with your own methods, direct input and output as needed, completion etc. http://apsw.googlecode.com/svn/publish/shell.html (Disclaimer: I am the APSW author) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktvyGMACgkQmOOfHg372QT5mgCgrMCtb3bHd3rF0L+lL/nZV6BX zrMAn1fcxS4CyKYF4KXVBcVcEXWhxoig =hpkY -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: SQLite3: preventing new file creation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gnarlodious wrote: > Every time I say something like: > > connection=sqlite3.connect(file) > > sqlite creates a new database file. Can this behavior be suppressed > through SQLite? Or am I forced to check for the file existing first? This is due to the API that pysqlite uses to talk to SQLite. (There is a more recent API but pysqlite remains backwards compatible with older SQLite versions). Note that although SQLite will create the file, it will be zero length (*) until you do a command that causes a database change. Also as a guideline be careful with SQLite files. In particular not only is there a database file, but there may also be a journal file. If the journal is removed then the main database file can be corrupted. (The journal contains data in order to rollback back incomplete transactions from the database.) (*) On Mac due to an operating system bug the file will actually be created as one byte in length containing the upper case letter 'S'. There is a dedicated mailing list for Python and SQLite: http://groups.google.com/group/python-sqlite You can use the newer SQLite database open API as well as many other SQLite APIs not supported by pysqlite by using APSW. (Disclaimer: I am the author of APSW.) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktvxyoACgkQmOOfHg372QQjqwCglx0u6OgGgOsQm0Bwd7s6BmCS 7EgAoKDdMZyDaw3Ov+Uqzs3RFX/NSHEK =/E0N -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Safe file I/O to shared file (or SQLite) from multi-threaded web server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Diez B. Roggisch wrote: > AFAIK, sqlite ensures process-serialization via locking, and threads > synchronize themselves as well. SQLite versions prior to 3.5 did not support using the same connection or cursors in different threads. (You needed to allocate, use, and close all in the same thread.) Since then SQLite objects can be used in any thread you want at any time. The SQLite error handling API is not threadsafe and requires a lock to be held otherwise you can get incorrect errors or worst case program crashes. The sqlite3/pysqlite code does not hold that lock (API introduced in SQLite 3.6.5) so you are only safe if you continue to only use objects in the same thread. If you use APSW then you can use any SQLite object at any time in any thread (it holds the lock amongst other things). > So you shouldn't need to worry at all. The main gotcha is that SQLite uses file locking and the default behaviour when unable to get a lock is to immediately return an error. SQLite does have an API to specify how long it should wait to acquire the lock (it keeps retrying until the time expires). sqlite3/pysqlite only lets you specify this maximum time when opening the connection and defaults to 5 seconds. On a busy server this may be too short so you'll end up getting busy errors. (Remember that writes require at least two disk syncs and that the default behaviour for Linux is to flush all outstanding writes not just for the file requested.) If you use APSW then you get default SQLite behaviour and two APIs - one lets you set/change the timeout period and the other lets you install your own busy handler which can do whatever it wants in order to prod things along. (Disclosure: I am the author of APSW.) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAks+17QACgkQmOOfHg372QSiCwCgpr6fSOr6UcUUZqTDoFA4RBcK zb8An21zZCr30AQ7VGP/Q/CsQ3z+2EVs =55MC -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: sqlite3 .mode option to create HTML table automatically?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 davidj411 wrote: > the CLI for sqlite3 shows .mode of "html", which formats the output in > HTML format that is good to add to . > > BUT i have not yet found anything for sqlite in python that does this. The CLI is extra code (in C) that wraps the SQLite library and adds functionality like this. You can call the CLI from your Python code (use the subprocess module) and capture output that way. Alternatively you can use APSW (disclosure: I am the author of APSW). APSW comes with a shell modelled on the SQLite one: http://apsw.googlecode.com/svn/publish/shell.html The Shell is written in Python (see Shell class lower in that page for API information). You can instantiate one with stdout set to a StringIO and then give it commands by calling shell.process_complete_line. You can also embed the Shell in your program, add commands and output modes etc. > also, i'd like to know if i can get headers in the result set. > The CLI uses '.header ON' to enable this. > is there a place where i can change this too? The APSW shell supports that. The APSW home page is at: http://code.google.com/p/apsw/ Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAks0QkIACgkQmOOfHg372QS9UgCeJh+QEx61AKK2+7XBFZ0hjIOu FtYAoKJdjlitJuFmix0DGq7hojqKTIpN =kRhG -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: sqlite question
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dennis Lee Bieber wrote: > I suspect, besides building an sqlite3.dll (if Windows), you might > have to modify the pysqlite DB-API adapter to support whatever new > arguments have been added to various calls (most like the db.connect() > parameters have changed to require an encryption password) The extension requires an extra C api call after the database is opened to set the encryption key. (There is also another API to change the key.) This means that it is not possible for the same pysqlite to work against SQLite built with and without the encryption extension. The changes needed in pysqlite are a simple matter of programming although you'll want them incorporated back into the core so you don't have to maintain them (wrapped in some sort of ifdef). Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrZb6sACgkQmOOfHg372QRSYwCg2J/YSvkqLs8EJ1iJkE2wnbaY nl0AoKYxgMEnBQNjDyYHv1xWC0Tia74U =eA2B -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: What command should be use when the testing of arguments is failed?
Peng Yu wrote: > I actually wanted to ask what return code should be returned in this > case when the arguments are not right. Thank you1 The BSD world attempted to standardize the codes so you may as well use their definitions. You can see them in /usr/include/sysexits.h on your nearest Linux/Unix box (usually). There is also an equivalent man page: http://www.freebsd.org/cgi/man.cgi?query=sysexits&sektion=3 The posix module also defines these constants if known for the platform. >>> import os >>> os.EX_USAGE 64 Doesn't help on non-Unix platforms though :-) Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: RabbitMQ vs ApacheQpid (AMQP)
jacopo wrote: > I am considering two solutions for a distributed system: either > RabbitMQ with py-amqplib or ApacheQpid with its own set of API. Have you considered the multiprocessing module? http://docs.python.org/library/multiprocessing.html#using-a-remote-manager Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: How to install 64-bit python on Ubuntu
Curious wrote: > Did you mean to say that Ubuntu does come pre-installed with 64-bit > Python? I am saying that 64 bit Ubuntu comes with 64 bit Python. (32 bit Ubuntu comes with 32 bit Python.) > When I used the same command as you did, I see a 32-bit > version there. It is most likely that you are running 32 bit Ubuntu (unless you bypassed Ubuntu's packaging and similar stunts). Run 'uname -a' and look near the end. For 64 bit you should see something like this: $ uname -m x86_64 If you see i686 instead of x86_64 you are on 32 bit. > $ file /usr/bin/python2.6 > /usr/bin/python2.6: ELF 32-bit LSB executable, Intel 80386, version 1 > (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, > stripped That is definitely a 32 bit binary. Try some others like /bin/ls and /sbin/init. (A 64 bit kernel can run 32 bit binaries but these core system files should all match the kernel.) > I also tried building Python on this machine by setting following > configuration option: > /configure --enable-universalsdk --with-universal-archs="64-bit" That is a road you don't want to go down unless you really know what you are doing and want to bypass the packaging system. As far as I am aware those options only apply to Macs anyway! > Could you please explain how did you get a 64-bit version there? When I download Ubuntu I got the 64 bit version. Look at the bottom of the page: http://www.ubuntu.com/getubuntu/download You may find various guides on how to convert a 32 bit installation into a 64 bit one. Do not do any of those (if you do then mention it when asking for help at which point people will tell you not to have done that!) Make a backup and do a *fresh* install. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: How to install 64-bit python on Ubuntu
Curious wrote: > Ubuntu comes pre-installed with Python2.6 but this python installation > is a 32 bit installation. For 64 bit Ubuntu you are mistaken: $ file /usr/bin/python2.6 /usr/bin/python2.6: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, stripped > I do the following to know if the current python installation is 32- > bit or 64-bit: This is what I do: >>> import ctypes >>> ctypes.sizeof(ctypes.c_void_p) 8 Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Combining python and sqlite DB into a single, "executeable".
[Please do not email me *and* the list - it is highly annoying] Tom Cumming wrote: > Thanks!, but I already thought of your suggestion. I've already gotten > the clear impression that the amount of work to implement this is more than > the ROI. It isn't anywhere near as hard or as much work as you think! > Having said that, it might work if on exiting the app could re-zip itself. See the atexit module. > One _big_ problem with zip files or compiled python executables is > moving the file > from machine to machine. It wouldn't be platform independent anymore. You are the one who mentioned executables which generally means platform specific binaries (and in the vast majority of cases that platform is Windows) :-) Being an executable means you don't have to care if Python is already installed. If you want it platform independent then you are stuck with Python text scripts. You can put one in a zip file and use PYTHONPATH or you can use the approach I showed at the bottom of my original message. > What I like best so far, is to have something like: You pretty much outlined what I wrote at the bottom of my message. > Convince the python parser not to go here... You can't do that. The entire source file has to be valid Python. You can get Python to not take action on parts by making it a multi-line string you do nothing with (ie surround with """) or by prefixing each line with # to make it a comment. I'd recommend a string at the end with base64 encoded contents. This would be very few lines of code to maintain along with tempfile.NamedTemporaryFile and atexit. My estimate is a total of 10-20 lines of code. > Now the trick is to get the sqlite library to start looking for the > sqlite database at, "offset". You cannot use the standard SQLite API to do that. SQLite does have a way of writing custom file access (known as VFS) that could do it and the APSW SQLite Python wrapper provides that functionality. See http://apsw.googlecode.com/svn/publish/vfs.html (Disclaimer: I am the author of APSW). However writing a VFS is *way* more work than something simple to extract data from the end of a script to a temporary file and put the data back at exit. And you'd have to ensure all machines have both Python and APSW installed. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Combining python and sqlite DB into a single, "executeable".
tcumming...@gmail.com wrote: > The problem, is that I need the python app and the > sqlite db file to exist in the same disk file. This way the app to > access the data and the data are in the same file. For binaries this is possible with a little hackery. Firstly you need make the app be a zip file. The good news is that zip files store their information at the end while executables store their information at the beginning. This is how self extracting zips work - they are the extractor program followed by the zip data. You can store the sqlite database in the zip portion. Copy it out to a tempfile while running and put it back in when done and saving. The tempfile module will help with temporary files. The zipfile module works quite happily with zip files prepended with extra stuff. To get started make a zip file with dummy contents (for example a readme explaining that mail is stored within). Assuming your app is called mail.exe and the zip file is called mail.zip you just concatenate them. py2exe and several other programs can make an executable out of a Python script for you. However in some cases they use this same 'trick' of an appended zip file to store the python code so you could also just modify that. If your app is a python script and you want to distribute it that way then you can make a zip file of the python script and then you have to run it by setting the PYTHONPATH environment variable to point at the zip file. Note that once you try this you may find the operating system and/or python not being too happy about modifying what they are "executing". If you are feeling really adventurous there may even be a way of following the script with the data. This is used in Unix shell scripting but the Unix shell doesn't check the whole file is syntactically correct so you can append any binary junk you want. For Python you could try something like 8< #! python python code python code """ SOMESPECIALMARKER <> """ 8< You'd have to do some escaping of the contents or maybe just base64 encode it. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Business issues regarding adapting Python
Nash wrote: > 3. If we do train people in Python for say a month; are we just > creating a team of mediocre programmers? Someone who has worked with > Python for over an year is much different than someone who has worked > with Python for only a month. In my experience the best way to "train" new developers is to have them work on porting, maintenance, bug fixing, testing etc of your product. This way they get exposed to your code, methodologies, quirks and values (eg security, internationalization, test coverage etc). It only needs to happen for a few months and has them in a position where they will do little harm to the main development. You'll also get a good idea where they will be best deployed. If your existing code base is a good example for the new developers to follow then this should work very well even for developers new to Python (but competent in other languages). > 4. Any suggestions or idea? Related posts, articles etc would > certainly help! Start a user group: http://wiki.python.org/moin/LocalUserGroups There is a Pakistan Linux User's Group and so should be some affinity and overlap with them. > I know that going Java will probably mean a 3x increase in the number > of people that we have and require time for Python component > replacement with Java ones. But for Business Continuity sake, > management doesn't mind. Or you could offer to pay Python developers more, and make it known that is happening. You'll soon find some more supply :-) Hopefully your next question will be about interviewing Python developers ... Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Signing extensions
Neil Hodgson wrote: >Code signing certificates that will be be valid for Windows > Authenticode cost $129 per year through CodeProject That isn't an amount I am prepared to pay either :-) (I don't even use Windows except as a glorified boot loader for Rise of Nations and to build Python extensions.) With the amount of hassle it causes me, I should be paid for the development time spent on Windows issues! >I'd like to see a certificate authority for open source projects > based mainly on project reputation and longevity. There may need to be > some payment to avoid flooding the CA with invalid requests - say $30 > per year. It would be great if this CA was recognised by Microsoft and > Apple as well as Linux and BSD distributions. It can also be solved as low down as Python itself, as opposed to open source in general. The Python installation could install a root CA for the PSF certifying authority although I suspect you can't then limit its use to only Python extensions. (I still find it amusing that the browser will silently accept certificates from any of the ~100 CAs that come with it. Your identity proof is only as strong as the weakest CA in the list, not the strongest.) It could also be solved by the download sites. For example Google Code does allow you to visit it via https and even displays the download page over https, but the downloads are over http. If it occurred to you then you can click on the "Summary+Labels" for an item where they show the SHA1 of the file, but that is even more hassle for most users. >There are some issues about identity here. You don't really need to worry about maliciousness. Ultimately that will come down to reputation. I am more concerned about download sites being hacked or malicious proxies being inserted into the network somewhere. It is good enough to be able to establish if this new version of the extension was produced by the same person as the previous version I have installed. PGP works wonderfully for that, except for Windows where no one has it. > The Ext1 project should be able to revoke ... That is pretty trivial to do if using regular CAs and OCSP. Of course someone still has to decide if the claim of maliciousness is correct or a joe job. Roger -- http://mail.python.org/mailman/listinfo/python-list
Signing extensions
I would like to digitally sign the open source Python extensions I produce. I produce source code (zip file) as well as pre-built binaries for Windows (all Python versions from 2.3 to 3.1). I can sign the source using my PGP key no problem. I could also sign the Windows binaries that way but Windows users are unlikely to have PGP and the Google code downloads page would look even worse having another 8 or 9 .asc files. The Windows Python distribution is signed by PGP and the normal Microsoft way using a Verisign class 3 cert. (If you read their issuer statement it ultimately says the cert isn't worth the bits it is printed on :-) One of those certs is $500 per year which is out of the question for me. Does anyone have any other suggestions? Has the PSF considered running a certificate authority for extension developers, and other Python developers for that matter? Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Do anyone here use Python *embedded* in a database?
Jonathan Fine wrote: > anyone here ever used the Python *embedded* in a database server. There is also the case of using SQLite where it shares the same process as your Python code (and nothing else) and is a standard part of the Python library. You can add your own functions and collations and if using APSW then you can also go way further than a regular database and add virtual tables and virtual storage. SQLite generally outperforms database servers since there is no cross process or network communication until you get to larger workloads and/or concurrent database access which database servers are more tuned for. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: hoe to build a patched socketmodule.c
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 jacopo mondi wrote: > Hi all, I need to patch socketmodule.c (the _socket module) in order to > add support to an experimental socket family. You may find it considerably easier to use ctypes since that will avoid the need for any patching. You'll also be able to control how read and write are done (eg read vs recvfrom vs recvmsg vs readv). You can use os.fdopen to convert your raw file descriptor into a Python file object if appropriate. If you do use ctypes then you'll only need to distribute pure Python source code. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkpWpCAACgkQmOOfHg372QQq0QCbB+rslUqK/AuUu0gA6N4m01Jo C8IAn3EkzFKCyt/K5WiuCnw3jzORBQh5 =JCv+ -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Opening a SQLite database in readonly mode
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joshua Kugler wrote: > BTW, APSW is written by the same author as pysqlite. Not even remotely true :-) pysqlite was written by various people, with the maintainer of the last several years being Gerhard Häring. I am the (sole) author of APSW and have not contributed any code to pysqlite although ideas have flowed freely between the projects and we share a mailing list. I started APSW in late 2004 because I wanted to use SQLite from Python rather than using a layer that pretended SQLite was like other databases. There were various quirks of pysqlite I also didn't like (many since corrected) and so scratched my itch. If you are just doing simple queries then there isn't much apparent difference. If you want to be a "power user" of SQLite then APSW is for you. SQLite has many cool features such as virtual tables (you provide the underlying data for the SQL queries to work on) and VFS (you provide the file access). See this link for more details: http://apsw.googlecode.com/svn/publish/pysqlite.html Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkpS6CUACgkQmOOfHg372QS0/gCgiHD9ukUlQYJGCIMWb9hNMLCM Y/cAnid4dAeFHIdLBmKzGsXrvANkvhR5 =U0wg -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Python preprosessor
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Tuomas Vesterinen wrote:
> I am intensively using 2to3.py. So I have 2 codebase: one in py2 and the
> other in py3.
The expectation would be that you only maintain the py2 code and
automatically generate the py3 code on demand using 2to3.
It is possible to have the same code run under both versions, but not
recommended. As an example I do this for the APSW test suite, mostly
because the test code deliberately explores various corner cases that
2to3 could not convert (nor should it try!). Here is how I do the whole
Unicode thing:
if py3: # defined earlier
UPREFIX=""
else:
UPREFIX="u"
def u(x): # use with raw strings
return eval(UPREFIX+"'''"+x+"'''")
# Example of use
u(r"\N${BLACK STAR}\u234")
You can pull similar stunts for bytes (I use buffers in py2), long ints
(needing L suffix in some py2 versions), the next() builtin from py3,
exec syntax differences etc. You can see more details in the first 120
lines of http://code.google.com/p/apsw/source/browse/apsw/trunk/tests.py
Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkosTxEACgkQmOOfHg372QS4rQCgl1ymNME2kdHTBUoc7/f2e+W6
cbMAmwf7mArr7hVA8k/US53JE59ChnIt
=pQ92
-END PGP SIGNATURE-
--
http://mail.python.org/mailman/listinfo/python-list
Re: Learning C++ for Python Development
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 joshua.pea...@gmail.com wrote: > Or, just give me some general advice on learning C++ for Python? You may want to start with Cython first. It lets you intersperse C and C level information with Python code to produce extensions. That will give you a gentler introduction to using a C like language with Python and for many real world uses is actually sufficient to get C levels of performance and interfacing with Python. After you get that working then you can try coding your own extensions (the Python docs include all the necessary information) and also trying other tools such as SWIG which automate some of the code for wrapping C++, or the wrapping functionality present in Boost. I find it most helpful to explore doing the same thing in multiple different ways as it teaches you what is important and what isn't. Later on it will help you a lot better in choosing the right tool for the job (no one tool or approach is perfect) and help evaluate anything that comes up later. My suggestion would be to pick a particular task and code it in pure Python. Benchmark it (cpu and memory) and then try the other approaches (Cython, Boost, hand coded, SWIG) and see how they compare. Something like a brute force Suduko solver would let you have a few classes/objects but not be too much code. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkoHy/QACgkQmOOfHg372QTxywCgrJPcHLQ5BzD8O29nSAW+I5xo juYAnjMP0xqn/TzC5mrTCqBT3ZnIQo24 =KeIk -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Unix Change Passwd Python CGI
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Derek Tracy wrote: > Apache is running on the same system that needs the password changed. I > need to keep security high and can not install additional modules at > this time. > > I just need a general direction to start looking, and I do not have > expect installed on the system. I recommend looking in the Samba source code where they have a program that does just that. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkmlGVcACgkQmOOfHg372QTZHACdFG0+Ls2Su/jRqkc4YZyxXK35 N7AAoNKfd7bMypR7b6Ex6auaU/9D4rKa =POal -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to increase memory allocated to the python interpreter
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 janandith jayawardena wrote: > Is there a way to configure the amount of memory allocated to the python > interpreter. Can it be increased or decreased using an argument like in > the Java Virtual Machine. Java needs the memory allocation number because of the way garbage collection is done (or at least used to be done). The CPython interpreter doesn't have an explicit limit and will use all the address space the operating system will let it have as needed. (That typically works out as around 2GB for a 32 bit process and exhausting swap space on a 64 bit process). If you want to prevent using more than a certain amount, then use functionality provided by your operating system. On Unix/Linux systems the ulimit command will do the trick. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkmlGMMACgkQmOOfHg372QQeMACfVhyccV91nU0WZswc2CNg8KMv SlEAoMINnO48FoDp0vgxROOWAjYp2tPG =Vjcf -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Fastest database solution
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Curt Hash wrote: > I started out using sqlite3, but was not satisfied with the performance > results. I then tried using psycopg2 with a local postgresql server, and > the performance got even worse. SQLite is in the same process. Communication with postgres is via another process so marshalling the traffic and context switches will impose overhead as you found. > I don't think > my code/queries are inherently slow, but I'm not a DBA or a very > accomplished Python developer, so I could be wrong. It doesn't sound like a database is the best solution to your issue anyway. A better solution would likely be some form of hashing the lines and storing something that gives quick hash lookups. The hash would have to do things like not care what variable names are used etc. There are already lots of plagiarism detectors out there so it may be more prudent using one of them, or at least learn how they do things so your own system could improve on them. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkmL/wgACgkQmOOfHg372QTAmACg0INMfUKA10Uc6UJwNhYhDeoV EKwAoKpDMRzr7GzCKeYxn93TU69nDx4X =4r01 -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Iterating through a file significantly slower when file has big buffer
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 pyt...@bdurham.com wrote: > The following tests were run on a Windows XP system using Python 2.6.1 Unless you changed the defaults, the Windows XP system cache size is 10MB. When you use a larger read size, chances are it is blowing out that cache and causes metadata (file block locations) to have to be reread on your next read. You are also funnelling all the data through your CPU cache with a similar effect although it will be less noticeable. To change XP cache sizes, see: http://marc.info/?l=sqlite-users&m=116743785223905&w=2 http://support.microsoft.com/kb/895932 http://www.techspot.com/tweaks/memory-winxp/ Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkl+4SkACgkQmOOfHg372QRgcACfVOdUWQGyj8xtNvHob/CtcM8g JsEAoKt/xI36iR5RuQOfZDMz2ze4L3Ia =DrDw -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: malloc (error code=12)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Arash Arfaee wrote: > Very BIG Jesse It works on a huge Boolean function. > And thanks Roger. Do you think it will be solved if I run it over > another OS like windows? By far the simplest solution is to use a 64 bit process on a 64 bit operating system. There are 64 bit versions of Windows available and you'll need to install the 64 bit version of Python (it has AMD64 in the name but will run on Intel 64 bit processors too). The various Linux distributions also have 64 bit flavours available. Note that you will also have to ensure that any 3rd party extensions using C code you use are also (re)compiled for 64 bit. The various versions of the 32 bit operating systems do have differing address space layouts but with large amounts of data in use you are still likely to hit a limit. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkl4K18ACgkQmOOfHg372QSE5QCgkp9bTPhkmHuh1I6U+onPLJSB 5MQAoIDmYNSlISe13Z2lSS0wspd/r+n+ =YQw6 -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: malloc (error code=12)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Arash Arfaee wrote: > Python(15492,0xb0103000) malloc: *** mmap(size=393216) failed (error > code=12) errno 12 is ENOMEM on Macs (and Linux for that matter). You may have run out of swap space, but that is unlikely. The most likely cause is that you have run out of address space. For example a 32 bit process has 4GB of address space available. The kernel will take up to 2GB of that depending on the operating system. Various shared libraries and your program will take some. The program stack space will take some more. In your case malloc is attempting to allocate 384kb of memory using mmap by mapping anonymous memory (swap) into the address space. (The advantage of doing it this way is that the space can be unmapped when no longer needed.) With no appropriate address space left for this chunk, you get the error. You are either trying to use too much stuff or there is a memory leak. Calling gc.collect() can result in some deferred items being freed. To get an idea of how address space works, check out this article about what Linux did in 2004. The same principles apply to MacOSX. You can also see how the theoretical 4GB quickly disappears! http://lwn.net/Articles/91829/ Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkl34YMACgkQmOOfHg372QRG9ACeKitN3f7cAcN1AOf76yTa9w+q CywAoL8JJ2/VvNe8eHOfP1r9UhOJH59z =EUjX -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Any news on when some libraries will be ported to Python 3.0?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Just Another Victim of the Ambient Morality wrote: > Anyway, I'd love to hear some news about any of these things in > particular or even anything in general. Am I the only one who's psyched for > this version of Python? I ported my APSW SQLite access module many months before Python 3.0 was released (around the time of the first beta release). The same codebase supports both Python 2 and 3 with very few conditionals and a few macros to make some Python 2 C api look like the Python 3 C api (eg pretending that the bytes type exists). Python 2.3 and up are supported on all platforms. It took longest to port my test suite over as I have 99.6% code coverage which is achieved by the test suite abusing every corner of the language and implementation. Fortunately a whole host of issues go away because of no 8 bit strings in Python 3. Other tools like Sphinx made my documentation so much better. That actually psyches me the most since Python is all about ease of reading and writing. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklkchwACgkQmOOfHg372QTgewCfXBrz0UGuUNDqxjVGEJOwfF4p N6QAoJvZCZ1Vm9f6sN0P4bXb8o3I2pVj =Lba+ -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Measuring bytes of packet sent from python application
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kangkook Jee wrote: > That seems like a good solution for my issue but how can I distinguish > traffics from my application to others? I use nethogs on Ubuntu. If you use Intrepid, you can press 'm' to make it change amongst different displays (eg cumulative traffic, send/receive rates etc). Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklizQYACgkQmOOfHg372QQc2gCgiMCeY1X9Qd/Fyh77SuDUNPqx 7lkAn1ozstsf2542+OuFuBC54vtOaaUR =qSxN -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: SQL, lite lite lite
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Aaron Brady wrote: > Python. There are some options, such as 'sqllite3', but they are not > easy. 'sqllite3' statements are valid SQL expressions, which afford > the entire power of SQL, but contrary to its name, it is not that > 'lite'. Have you compared the compiled size of SQLite against other things? For example on my machine the size of MySQL client library, whose sole purpose is to transport queries and results across the network is the same size as the entirety of SQLite! You can prune SQLite back even further as documented in http://www.sqlite.org/compile.html It is even possible to omit the SQL front end. Queries are stored already processed in the database. This functionality is used by mp3 manufacturers and similar constrained embedded environments. > To me, 'lite' is something you could learn (even make!) in an > afternoon, If you just want to treat the database as a glorified spreadsheet then SQL is "lite", although perhaps a little verbose of a dbm style interface. > If you think SQL is > a breeze, you probably won't find my idea exciting. I assume that the > basics of SQL are creating tables, selecting records, and updating > records. The basics of SQL are about expressing the relational model http://en.wikipedia.org/wiki/Relational_model which has stood the test of time. (That doesn't mean it is superior just that it is good enough like the "qwerty" keyboard layout.) There have been attempts at alternatives like http://en.wikipedia.org/wiki/The_Third_Manifesto but that doesn't seem to have caught on. It seems your basic complaint is the complexity of doing database stuff. Ultimately this will be the case if some data is related to other bits of data. As other posters have pointed out, there are various ORM type wrappers for Python that try to wrap this up in syntactic sugar :-) For something completely different have a look at CouchDB http://couchdb.apache.org/ which operates on "documents" (basically something with an id and an arbitrary updateable list of properties). It does take a bit to get your head wrapped around it - try this posting for an example http://www.cmlenz.net/archives/2007/10/couchdb-joins Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklZRp0ACgkQmOOfHg372QQ4RQCgzSmgEhvG2DQlWYb68U8BZNOo wOAAnip4GIvKiskuwwWJwaepmJwHLjXJ =0UiA -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Most efficient way to build very large dictionaries
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Martin wrote: > I'd think he's talking about in memory SQLite Databases, this way you > should be quite fast _and_ could dump all that to a persistent > storage... I was just talking about regular on disk SQLite databases. In terms of priming the pump, you can just open/read/close the whole database file which will cause the database to be in the operating system cache buffers, or just let SQLite do its thing. For anyone who is curious about what Martin is referring to, SQLite does support the database file being memory (although it isn't a file at that point). It has a fake filename of :memory:. As an example you can copy the table 'example' to memory using: ATTACH ':memory:' as memdb; CREATE TABLE memdb.memexample AS select * from example; As with Python, all this stuff is premature optimization. Get it working right first and then try tweaks to improve performance. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklSEqcACgkQmOOfHg372QTPpgCgvSKGMCJAKhnm5I8qHdmZtRh3 SgMAoI3DVhWCVdUE1TLck9ZEfp/Ln1H5 =5kNT -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Most efficient way to build very large dictionaries
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 pyt...@bdurham.com wrote: > Thank you for your suggestion about looking at SQLite. I haven't > compared the performance of SQLite to Python dictionaries, but I'm > skeptical that SQLite would be faster than in-memory Python dictionaries > for the type of analysis I'm doing. I'd recommend at least trying a test just to see. As an example SQLite uses indices which will be faster than Python dicts for some set operations. (And if you aren't careful, your various Python based optimizations will end up duplicating what SQLite does internally anyway :-) > Prior to my use of Python, my > customer used a very expensive Oracle system to analyze their log files. > My simple Python scripts are 4-20x faster than the Oracle PL/SQL they > are replacing - and run on much cheaper hardware. SQLite is not like Oracle or any similar database system. It does not operate over the network or similar connection. It is a library in your process that has an optimized disk storage format (single file) and a SQL parser that generates bytecode for a special purpose virtual machine in pretty much the same way CPython operates. The performance improvements you are seeing with Python over Oracle are exactly the same range people see with SQLite over Oracle. One common usage reported on the SQLite mailing list is people copying data out of Oracle and running their analysis in SQLite because of the performance advantages. > Note: Memory is currently not a concern for me so I don't need SQLite's > ability to work with data sets larger than my physical memory. The pragmas tune things like cache sizes. The SQLite default is 2MB, relying on the operating system for caching beyond that. Bumping up that kind of size was my suggestion :-) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklR9+UACgkQmOOfHg372QSMbwCdGS5S2/96fWW8knjfBVqReAfV AEwAn2Yc+L9BEZgT69OjwtyqxLtifVpU =mPfy -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Most efficient way to build very large dictionaries
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 pyt...@bdurham.com wrote: > Can I take advantage of this knowledge to optimize You do the optimization last :-) The first thing you need to do is make sure you have a way of validating you got the correct results. With 25M entries it would be very easy for an optimization to get the wrong results (maybe only one result wrong). Once you can verify the results correctness, write the simplest most readable code possible. Then once your whole program is approaching completion time how long things take to get an idea of where to optimize. For example it is pointless optimizing the loading phase if it only takes 2 minutes out of a 3 hour runtime. Finally you can optimize and always have a way of verifying that the optimizations are correct. You have the original code to document what is supposed to be happening as optimized code tends to get rather obfuscated and unreadable. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklR8mkACgkQmOOfHg372QQvLQCgu6NYNUuhgR06KQunPmIrZ64B +rsAnAgQOKzMdmonF+zIhsX2r/Xg/72Y =LFfW -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Most efficient way to build very large dictionaries
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 pyt...@bdurham.com wrote: > I would appreciate your thoughts on whether there are advantages to > working with a pre-built dictionary and if so, what are the best ways to > create a pre-loaded dictionary. Based on this and your other thread, you may want to consider using SQLite (standard Python module is available for it). SQL queries are very similar to set operations and indices take care of performance (by using more storage). You also get transactions for free. If you look into SQLite pragmas you can get it to use more RAM to improve performance. And by using a SQL layer you can later switch to another database should you need really hard core storage. It also makes the data available to other programs in a uniform way. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklR8KgACgkQmOOfHg372QSrHQCfVJzueXVKme8QZcxoLf70BL4K RL8AoM9QOFykOLrr5QXtpmZ5f7CFHm6e =zAPG -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Strategy for determing difference between 2 very large dictionaries
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 pyt...@bdurham.com wrote: > Feedback on my proposed strategies (or better strategies) would be > greatly appreciated. Both strategies will work but I'd recommend the second approach since it uses already tested code written by other people - the chances of it being wrong are far lower than new code. You also neglected to mention what your concerns are or even what "very large" is. Example concerns are memory consumption, cpu consumption, testability, utility of output (eg as a generator getting each result on demand or a single list with complete results). Some people will think a few hundred entries is large. My idea of large is a working set larger than my workstation's 6GB of memory :-) In general the Pythonic approach is: 1 - Get the correct result 2 - Simple code (developer time is precious) 3 - Optimise for your data and environment Step 3 is usually not needed. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklR5DUACgkQmOOfHg372QSuWACgp0xrdpW+NSB6qqCM3oBY2e/I LIEAn080VgNvmEYj47Mm7BtV69J1GwXN =MKLl -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Jarow-Winkler algorithm: Measuring similarity between strings
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Øyvind wrote: > Based on examples and formulas from http://en.wikipedia.org/wiki/Jaro-Winkler. > Useful for measuring similarity between two strings. For example if > you want to detect that the user did a typo. Jaro-Winkler is best when dealing with names (Winkler works for the US census). There are pure Python and C accelerated implementations at http://bitpim.svn.sourceforge.net/viewvc/bitpim/trunk/bitpim/src/native/strings/ If you are concerned about typos then taking into account the keyboard layout will help. For example for a user with a US keyboard, the 'a' or 'd' keys would be a common typo for 's'. Also consider Levenshtein distance: http://en.wikibooks.org/wiki/Algorithm_implementation/Strings/Levenshtein_distance Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklMUEkACgkQmOOfHg372QRTlQCfUoebzX2HRbQ4wLVZ6yRFMHd7 9yMAnjovqefVuQenX0zpHwn/rvv9FLe+ =bACc -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Source code generation using Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ats wrote: > I want to generate 3 different versions of a C++ source code, > basically injecting different flavours of inline assembler depending > on target compiler/CPU. Are you aware that there are also packages that let you generate and call C code from Python on the fly? I find it most productive to write my code in all Python first and to also develop a comprehensive test suite. Then profile and replace selected portions with lower level C code with the tests being able to confirm your code is correct. Here are some packages that take an alternate approach: http://www.cs.tut.fi/~ask/cinpy/ http://code.google.com/p/shedskin/ http://pyinline.sourceforge.net/ http://scipy.org/Weave http://mdevan.nfshost.com/llvm-py/ I like LLVM the most. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkk8MkUACgkQmOOfHg372QRhsgCcCUzWHAHmjC1490yYba7c9Xrt DxMAnj/Ur2GoJkQgMrx65hYEqPwKLdVV =CvGB -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: Determining number of dict key collisions in a dictionary
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: > Background: I'm working on a project using very large dictionaries (64 > bit Python) and question from my client is how effective is Python's > default hash technique for our data set? Python hash functions return a long which in a 64 bit process is 32 bits on Windows and 64 bits on pretty much every other 64 bit environment. > Their concern is based on the > belief that Python's default dictionary hash scheme is optimized for 32 > bit vs. 64 bit environments and may not have anticipated the additional > range of keys that can be generated in a 64 bit environment. Our keys > are based on 20 to 44 byte ASCII (7-bit) alpha-numeric strings. Why not have them look at the source code? It is well commented and there is another file with various notes. Look at Objects/dictobject.c and Objects/dictnotes.txt A teaser comment for you: Most hash schemes depend on having a "good" hash function, in the sense of simulating randomness. Python doesn't. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkk1jsUACgkQmOOfHg372QTeEQCeJwkRphiPeDefkANg1IdG3HH1 oocAoICJk6NGxVmtZTZtLOL4Sv4aCw1n =IqsO -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: More than one interpreter per process?
Graham Dumpleton wrote:
> When using mod_wsgi there is no problem with C extension modules which
> use simplified GIL API provided that one configures mod_wsgi to
> delegate that specific application to run in the context of the first
> interpreter instance created by Python.
Graham, I've asked you before but never quite got a straight answer.
What *exactly* should extension authors change their code to in order to
be fully compatible? For example how should the C function below be
changed:
void somefunc(void)
{
PyGILState_STATE gilstate=PyGILState_Ensure();
abc();
Py_BEGIN_ALLOW_THREADS
def();
Py_END_ALLOW_THREADS
ghi();
PyGILState_Release(gilstate);
}
Roger
--
http://mail.python.org/mailman/listinfo/python-list
Re: More than one interpreter per process?
sturlamolden wrote: > On 18 Des, 10:24, Roger Binns <[EMAIL PROTECTED]> wrote: > >> The biggest stumbling block is what to do when the external environment >> makes a new thread and then eventually calls back into Python. It is >> hard to know which interpretter that callback should go to. > > Not if you explicitely hav to pass a pointer to the interpreter in > every API call, which is what I suggested. You missed my point. What if the code calling back into Python doesn't know which interpreter it belongs to? Think of web server with python callbacks registered for handling various things. Currently that situation works just fine as the simplified gil apis just pick the main interpreter. You have now imposed a requirement on all extension modules that they need to keep track of interpreters in such a way that callbacks from new threads not started by Python know which interpreter they belong to. This is usually possible because you can give callback data in the external environment apis, but your mechanism would prevent any that don't have that ability from working at all. We wouldn't find those "broken" implementations until changing to your mechanism. > But porting could be automated by a simple Python script. Have you actually tried it? See if you can do it for the sqlite module which is a standard part of the Python library. > PEPs are closed for Python 3. You glossed over my "prove the benefit outweighs the costs" bit :-) This project will let you transparently use multiple processes: http://cheeseshop.python.org/pypi/processing There are other techniques for parallelization using multiple processes and even the network. For example: http://www.artima.com/forums/flat.jsp?forum=106&thread=214303 http://www.artima.com/weblogs/viewpost.jsp?thread=214235 Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: More than one interpreter per process?
sturlamolden wrote: > If one can have more than one interpreter in a single process, You can. Have a look at mod_python and mod_wsgi which does exactly this. But extension modules that use the simplified GIL api don't work with them (well, if at all). > Most of the conversion of the current Python C API could be automated. The biggest stumbling block is what to do when the external environment makes a new thread and then eventually calls back into Python. It is hard to know which interpretter that callback should go to. You are also asking for every extension module to have to be changed. The vast majority are not part of the Python source tree and would also have to support the versions before a change like this. You would have more luck getting this sort of change into Python 3 since that requires most extension modules to be modified a bit (eg to deal with string and unicode issues). But before doing that, why not show how much better your scheme would make things. The costs of doing it are understood, but what are the benefits in terms of cpu consumption, memory consumption, OS responsiveness, cache utilisation, multi-core utilisation etc. If the answer is 1% then that is noise. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: How to release the GIL from C?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roger Binns wrote: > I am trying to release the GIL in a multi-threaded program (efforts > detailed below) without any success. The ultimate cause was that the program forked to go into daemon mode. I had called PyOS_AfterFork() as the documents directed. What I hadn't realised (and isn't documented) is that AfterFork() reinitialises the Python threading state by making a new GIL and acquires it. The acquiring bit meant none of my other threads could ever get it. It is still unclear from the docs exactly which combination of functions dealing with threadstate, threads and lock you need to call to just unilaterally give up the GIL (ie when you don't have following block that wants to reacquire the GIL) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFGVNW0mOOfHg372QQRAuyOAJ4pCCIK779XgvaUdKBtSa+nHElrHQCgiueP n/0uMFCSH3SrQhMXdm2Jb/o= =uh4D -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
How to release the GIL from C?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I am trying to release the GIL in a multi-threaded program (efforts
detailed below) without any success.
In the main thread during startup, I do the following:
Py_InitializeEx(0); /* Python shouldn't handle signals */
PyEval_InitThreads();
/* release the GIL */
... various pieces of code tried here ...
Note that no other threads exist yet when the code above is run.
After startup, there are around 30 threads and my handler routines do
work in them.
void handler(...)
{
PyGILState_STATE gilstate=PyGILState_Ensure();
... do the interesting stuff ...
PyGILState_Release(gilstate);
}
No matter what I try, the PyGILState_Ensure() hangs trying to acquire
the GIL:
#0 0x2afd50ce243d in sem_wait () from /lib/libpthread.so.0
#1 0x2afd5471557d in PyThread_acquire_lock () from
/usr/lib/libpython2.5.so.1.0
#2 0x2afd546e44d4 in PyEval_RestoreThread () from
/usr/lib/libpython2.5.so.1.0
#3 0x2afd5470bb8a in PyGILState_Ensure () from
/usr/lib/libpython2.5.so.1.0
Here are the various things I have tried to release the GIL (the
documentation says they all do, but that doesn't appear to be the case
:-( Unfortunately the doc is all geared around saving the interpreter
state and then later restoring it. I have nowhere to save it in the
main thread nor do I know when the main thread is having handlers
invoked in it again anyway in order to restore it.
1. (Based on Py_BEGIN_ALLOW_THREADS)
PyThreadState_Swap(NULL);
PyEval_ReleaseLock();
2. (Based on high level Py_BEGIN_ALLOW_THREADS)
PyEval_SaveThread();
3. Random code I found on the web
PyThreadState *tstate = PyThreadState_Get();
PyThreadState_Clear(tstate);
PyEval_ReleaseThread(tstate);
4. Something simple
PyEval_ReleaseLock();
5. Other code I found on the web
PyThreadState_Swap(NULL);
Does anyone know exactly which sequence of calls I should be making?
Thanks,
Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFGVKKSmOOfHg372QQRAi4VAJ9KO8z/WnSgPOlCjEOOJgDCf20nCwCeJHUE
kkhvMsy6p3qflvGKolttWTo=
=4Z8h
-END PGP SIGNATURE-
--
http://mail.python.org/mailman/listinfo/python-list
Adding extra frames to traceback in C module
One thing I would like to do in my extension module is add extra frames to the traceback when an extension occurs. At the moment C code is invisible to tracebacks. This is relevant when the C code makes a Python callback. For example if the following code sequence happens (time going down) Python C A B C D E F G H If there is an exception in H then the traceback seen from A will be "B C G H". I want to get D, E and F into there as well. This is because "G H" could be called for several different reasons in the C code and adding the extra information will establish why. I couldn't find anything on the web or in the documentation to explain how to do it. I did find snippets of code doing things like PyTraceback_Here but they use a real Python frame which I don't have and don't know how to synthesize. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Help on exceptions (was: Convertion of Unicode to ASCII NIGHTMARE)
"Serge Orlov" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > It can be like this. Notice special url, it is pointing to a > non-existing :) tutorial about why concatenating byte strings with > unicode strings can produce UnicodeDecodeError An alternate is to give an error code that people can use to look up (and users can report). It also has the advantage of being language neutral. You can see this kind of approach on IBM systems (mainframe, AIX etc) and Oracle. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Insertion (sql) bug in Py2.4 pySQLite 2.2
"DurumDara" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > I have this code in my program. Before this I use APSW, but that project's > connection object doesn't have close method... The connection object is released when there are no more references to it, and there are no outstanding cursors. The existence of the connection object does not hold a database open. In fact the only thing that holds a database open is cursors inside a transaction. And if those exist, calling an explicit close on the connection object would merely generate an error message that cursors etc exist. I recommend reading this document: http://www.catb.org/~esr/faqs/smart-questions.html Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Convertion of Unicode to ASCII NIGHTMARE
"Serge Orlov" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > I have an impression that handling/production of byte order marks is > pretty clear: they are produced/consumed only by two codecs: utf-16 and > utf-8-sig. What is not clear? Are you talking about the C APIs in Python/SQLite (that is what I have been discussing) or the language level? At the C level, SQLite doesn't accept boms. You have to provide UTF-8 or host byte order two bytes per char UTF-16. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Convertion of Unicode to ASCII NIGHTMARE
"Fredrik Lundh" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > sounds like your understanding of Unicode and Python's Unicode system > is a bit unclear. Err, no. Relaying unicode data between two disparate C APIs requires being careful and thorough. That means paying attention to when conversions happen, byte ordering (compile time) and boms (run time) and when the API documentation isn't thorough, verifying the behaviour yourself. That requires a very clear understanding of Unicode in order to do the requisite test cases, as well as reading what the code does. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Convertion of Unicode to ASCII NIGHTMARE
"Fredrik Lundh" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Roger Binns wrote: > >> SQLite only accepts Unicode so a Unicode string has to be supplied. > > fact or FUD? let's see: Note I said SQLite. For APIs that take/give strings, you can either supply/get a UTF-8 encoded sequence of bytes, or two bytes per character host byte order sequence. Any wrapper of SQLite that doesn't do Unicode in/out is seriously breaking things. I ended up using the UTF-8 versions of the API as Python can't quite make its mind up how to represent Unicode strings at the C api level. You can have two bytes per char or four, and the handling/production of byte order markers isn't that clear either. >import pysqlite2.dbapi2 as DB pysqlite had several unicode problems in the past. It has since been cleaned up as you saw. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Convertion of Unicode to ASCII NIGHTMARE
"ChaosKCW" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > me. As for SQLite supporting unicode, it probably does, No, SQLite *ONLY* supports Unicode. It will *only* accept strings in Unicode and only produces strings in Unicode. All the functionality built into SQLite such as comparison operators operate only on Unicode strings. > but something > on the python side (probabyl in apsw) converts it to ascii at some > point before its handed to SQLite. No. APSW converts it *to* Unicode. SQLite only accepts Unicode so a Unicode string has to be supplied. If you supply a non-Unicode string then conversion has to happen. APSW asks Python to supply the string in Unicode. If Python can't do that (eg it doesn't know the encoding) then you get an error. I strongly recommend reading this: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets http://www.joelonsoftware.com/articles/Unicode.html > Ok if SQLite uses unicode internally why do you need to ignore > everything greater than 127, I never said that. I said that a special case is made so that if the string you supply only contains ASCII characters (ie <=127) then the ASCII string is converted to Unicode. (In fact it is valid UTF-8 hence the shortcut). > the ascii table (256 bit one) fits into > unicode just fine as far as I recall? No, ASCII characters have defined Unicode codepoints. The ASCII character number just happens to be the same as the Unicode codepoints. But there are only 127 ASCII characters. > Or did I miss the boat here ? For bytes greater than 127, what character set is used? There are hundreds of character sets that define those characters. You have to tell the computer which one to use. See the Unicode article referenced above. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Convertion of Unicode to ASCII NIGHTMARE
"Paul Boddie" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > It looks like you may have Unicode objects that you're presenting to > sqlite. In any case, with earlier versions of pysqlite that I've used, > you need to connect with a special unicode_results parameter, He is using apsw. apsw correctly handles unicode. In fact it won't accept a str with bytes >127 as they will be an unknown encoding and SQLite only uses Unicode internally. It does have a blob type using buffer for situations where binary data needs to be stored. pysqlite's mishandling of Unicode is one of the things that drove me to writing apsw in the first place. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: State of SSL in Python
<[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > I'd like to use Python's native SSL functions because I'd like to keep > the install requirements at a minimum. I'm writing a client that will > use TLS with X509 certificate validation (and CRL checking in the > future). Will Python be able to handle it? Try TLSLite: http://trevp.net/tlslite/ Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Python source cross reference doc generator?
"Harry Fuecks" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Wondering if a tool exists to generate "cross reference" documentation > for Python code bases? PyXR does cross referencing. epydoc generates good doc from comments (javadoc style): http://pyxr.sourceforge.net/ http://epydoc.sourceforge.net/ This their output run over a non-trivial codebase: http://www.bitpim.org/pyxr/ http://www.bitpim.org/apidoc/ It would be really nice if they were both integrated together. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Which GUI toolkit is THE best?
"invitro81" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > But I've no idea which one I should use to start with.. One thing you'll need to carefully decide is where you want to end up. The different toolkits have different limits on where you can go. A simple example is printing. Some don't support it (eg Tkinter didn't 5 years ago when I last used it). If you will never need to print then that won't matter. Generally what you'll find is that the toolkits that let you do more are harder to get started with because you have to code in such a way as to let the more complicated features (eg printing) work, and the documentation is bigger because there is more to document. These are some of those areas to consider: Printing Can you print? Does the user get to use their operating system specific dialog boxes to choose printers etc? As a developer can you find out if the printer is colour or black and white? What about the paper size? HTML Does the toolkit have a way for you to display HTML? How complicated can the HTML be (eg CSS)? Can you embed widgets in the HTML? Threading Do you have to confine gui operations to one thread? How do you send a message from a worker thread to the gui thread (the gui thread will block in an event loop - that event loop needs to be able to be woken up from other threads) Native look and feel Will your users expect a native looking application? Will they want drag and drop? Do they expect rich objects on the keyboard? Is the toolkit available for all platforms you would want to run on (Windows, Linux, Mac, BSD etc) and their variations (different processors, 32 and 64 bit runtimes)? OLE Can you display other applications as a widget within yours? What about the other way round? Can you script other applications? (Python actually has the latter available on many platforms without the need for the gui toolkit to so) i18n/l10n Does the toolkit support Unicode? How easy is it to use with multiple languages? Additional packages What additional packages are there for the toolkit? For example Pmw provides more widgets to tkinter and their are GL packages for the various toolkits. If you are going to do graphics heavy stuff you'll want some way of hooking the toolkit with PIL. There should be some sort of demo available. wxPython has a fantastic one showing every single widget with code you edit to see what effect it has. The documentation should be good as well. Pick a random question such as "how do I delete all entries in a listbox begining with 'foo'" and see if you can work it out from the documentation. There should also be some sort of mailing list and user forums. Look for helpful answers being given with people being polite. What I would suggest you do is write a simple application in two or three different toolkits (eg convert temperature between F and C). You'll learn various ways of dealing with widget placement and sizers. (ie what happens if the user makes the containing window larger or smaller). My suggestions are to try PyGame, Tkinter and wxPython. You'll be better off having learned lessons from all 3 than just using one. The piece of good news is that to my knowledge applications written using any of the toolkits can be packaged up using tools like py2exe/py2app/cx_Freeze so that you can redistribute them to other users and those users will not have to know or care that you used Python and whatever toolkit. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Distributed Cache Server?
<[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Does anyone know if a "distributed caching system" has been developed > for use with Python? BitTorrent :-) > Yes, "distributed caching system" is a bit of a general term, but am > really just talking about something as simple as key + value (arbitrary > class) which can be split over a number of machines in an efficient > manner. You'll need to define what the sweet spot is that you are aiming for. Are we talking tens of thousands of keys or billions? How big is the data (megabytes, gigabytes, terabytes?) Do you need transactional integrity (eg when are updates seen by other readers)? Do you want redundancy (data duplicated on multiple machines)? How many machines are we talking about? Should failure be automatically detected? Is there a need for security or treating the machines as untrusted? Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: PyFLTK - an underrated gem for GUI projects
"aum" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > To me, wxPython is like a 12-cylinder Hummer, with fuzzy dice hanging > from the mirror, fridge and microwave in the back, and DVD consoles on > every seat, towing a campervan - absolute power and luxury, giving 8mpg > if you're lucky. A word of support for wxPython: - It comes with a nice demo app that shows every single supported widget and editable code for playing with them so you can tweak to try new things. - It supports printing, drag and drop and various similar niceties on each platform. - Mac is a first class citizen (support isn't perfect but it is better than "good enough"). The FLTK docs mention Mac a few times but usually just Unix and Windows. - It fully supports Unicode. (It looks like the under development FLTK 2.0 uses UTF-8 in some places but doesn't look like this release is imminent.) - There is a half decent HTML widget. (The FLTK one is only a quarter decent :-) - You get the native look and feel on each platform and native widgets are used wherever possible. (I can't tell what FLTK does). - Various other corner things are covered as needed by more complex apps such as audio, ActiveX on Windows, stock icons, online help, calendar controls, directory and file selectors, a grid/table etc etc As is usually the case, each developer only uses 10% of the functionality of the toolkit available, but it is a different 10% for each! My 10% includes printing and drag and drop which are missing from most of the toolkits you listed. I also insist on native widgets wherever possible. It is possible to make the wxPython smaller by having more DLLs each with fewer widgets in it. That way you will only suck in the ones you use. I do like that FLTK has the documentation style as PHP where users can add comments to each page. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Generating HTML from python
"Philippe C. Martin" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Hi, > > I wish to use an easy way to generate reports from wxPython and feel > wxHtmlEasyPrinting could be a good solution. > > I now need to generate the HTML wxHtmlEasyPrinting can print: I need to have > a title followed by lines of text that do not look too ugly. If possible I > would like to use an existing module. > > Q1) Is there such a module ? > Q2) Is my approach fairly good ? There is one huge problem with wxHtml and consequently wxHtmlEasyPrinting. It doesn't understand stylesheets in any way. There are many packages that generate HTML, but they all expect a lot of the styling to be handled by CSS. It is also a lot easier to write that way. I would absolutely love a piece of code that could take CSS and HTML and apply the CSS to the HTML to get the right appearance, but without any CSS left. For example it would turn CSS formatting for class/div/span into HTML attributes, as well as overal formatting instructions. I did do something similar for BitPim but it requires the HTML to be perfect and the CSS is actually a Python data structure expressing the attributes and how they get applied. I'd love to replace it with something better. But ultimately it does work. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparing 2 similar strings?
<[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > ***Check out difflib, it's in the library.*** Perfect package for what > the OP wants AFAICT. The method in difflib is okay, but doesn't do that good a job. It is also relatively slow. My need for this was matching records in BitPim (eg which entries just read from a cell phone correspond to which entries just read from Outlook) Here are some links for the OP as well as ready to run code at the end. A paper on various algorithms and their effectiveness on various data sets. http://www.isi.edu/info-agents/workshops/ijcai03/papers/Cohen-p.pdf They do have a Java library of all the algorithms by the same authors. http://secondstring.sourceforge.net/ There is a group at the Australian National University with a project named Febrl - Freely extensible biomedical record linkage. The idea is to be able to work out if two records from the same or different sources are the same person. All their code is in Python: http://datamining.anu.edu.au/software/febrl/febrldoc/ The solution I settled on is using Jaro-Winkler. This is a quote from the paper above: A surprisingly good distance metric is a fast heuristic scheme, proposed by Jaro (Jaro 1995; 1989) and later extended by Winkler (Winkler 1999). This works almost as well as the Monge-Elkan scheme, but is an order of magnitude faster I did find a Python module that implemented this (it is part of the python-levenshtein package). Unfortunately there were some issues at the time dealing with some strings that may be unicode and some not. There was one implementation I saw that took copies of both strings and was replacing characters with asterisks. My data sets include meaningful asterisks so that seriously munged things. Ultimately I wrote my own implementation with the following properties: - The arguments can be any combination of unicode and ordinary strings - There are no special characters nor ones used internally as replacements - A bitset is used for tracking so you use an eigth of the memory - There is both Python and C code implementations so you don't have to compile anything but can if you want The code can be grabbed from: http://cvs.sf.net/viewcvs.py/bitpim/bitpim/native/strings/ There is a detailed description at the begining of the Python implementation: http://cvs.sf.net/viewcvs.py/bitpim/bitpim/native/strings/jarowpy.py?view=markup Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: pyvm -- faster python
"Stelios Xanthakis" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] >> - hacking SWIG. Shouldn't be too hard and will instantly give >> us access to wx, qt, etc. Have you ever written a non-trivial extension using Swig? It isn't as simple as you would think. There are a lot of little things to deal with, usually because because of the mismatch between C/C++ semantics and convention and Python library expectations. You have to do a fair amount of work to fix all that using C code, Python code and sometimes parts of the internals of Swig-Python. wxPython has used a modified version of Swig for most of its lifetime. Some people also have issues with Swig staying compatible with itself: http://bob.pythonmac.org/archives/2005/03/11/swig-hate/ PyQT uses another tool to do the wrapping - SIP. Tools like Swig have been in the business of exposing C/C++ to Python for years and are still suboptimal. I believe you will find "hacking SWIG" and actually porting an existing project that uses it to be hard. I would suggest looking in the lib/python directory of Swig to get an idea of what you have to start with. Note how there is minimal overlap with the Java directory to pick one example. >> The thing is that the C API of pyvm is IMHO superior and much more fun. That is not under dispute. The suggestion is that you *also* provide a way of running existing extensions, even if it is slow and inferior, ideally without even having to recompile them. (If you do require a recompile then you'll also have to port/invent an equivalent to distutils). >> You can wrap the entire sockets module in a couple of hours and also >> enjoy it. sockets are easy. There are no structures, plain integer handles, no pointers, about 8 methods and a few constants. About the only thing to remember is to deal with threading correctly. >> I wish I could clone myself to port the entire std library >> to pyvm -- so much fun it is:) Well, if you don't provide an easy way to move existing stuff across, you'll end up doing just that yourself! Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: pyvm -- faster python
"Paul Rubin" <http://[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > "Roger Binns" <[EMAIL PROTECTED]> writes: > What kind of stuff is in the existing Python C library that couldn't > be reimplemented or retargeted pretty easily? Most of it is either > wrappers for standard C functions (system calls), stuff that shouldn't > be in C in the first place (the Python compiler and interpreter), etc. Some examples are gui toolkits (eg wxPython), SSL (eg M2Crypto, pyopenssl) and database (pysqlite, APSW). These aren't in the shipped with Python library but are widely used. > I hope that PyPy will replace CPython once it's solid enough. Trying > to stay backwards compatible with the legacy C API doesn't seem to me > to be that important a goal. Redoing the library may take more work > than the Prothon guy was willing to do for Prothon, but PyPy has more > community interest and maybe can attract more resources. You don't have to stay backwards compatible. It is best to provide some sort of way of using the old extensions even if it is suboptimal (eg some sort of mapping shim). The extensions can later be improved, but if there is no simple way of recompiling and running then it will take a long time for all the extensions to get ported. I already get burnt out on the matrix of CPython versions and different platforms. Adding another interpretter would make life even harder for extension authors. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: pyvm -- faster python
"Paul Rubin" <http://[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > "Roger Binns" <[EMAIL PROTECTED]> writes: >> That will rule out all the gui frameworks, SSL, cryptography >> and numerous other packages. Have a look at what happened to >> Prothon. > > I think it would be enough to retarget SWIG. That is also a lot of work, and a lot of toolkits that use Swig for Python bindings are very strongly tied to the Swig-Python internals. For example I doubt there are many where you could tell Swig to output Java and it would just work. (I know none of mine would work, nor would wxPython). >> What ultimately killed it was the problem of having a decent >> library. > > Nah, I think people just lost interest. Otherwise, why isn't Jython > dead? Err, you proved my point! Prothon was fine at the VM level. The author couldn't figure out how to get a decent sized "standard" library, and ultimately ended up abandoning his VM for .Net since that gets you a large standard library. Jython also gets a large standard library from the Java ones. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: pyvm -- faster python
> I am not very interested on C compatibility. That will rule out all the gui frameworks, SSL, cryptography and numerous other packages. Have a look at what happened to Prothon. What ultimately killed it was the problem of having a decent library. You don't have to make the C library compatibility totally high performance, but having any form of it there will make adoption by others easier. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: M2Crypto SSL memory leaks - fixes or alternatives ??
> I notice that M2Crypto (a python wrap of OpenSSL) leaks (haemorrhages) > memory significantly and affects my long running app very badly. > > Does anyone know of fixes to this problem? > > Does anyone recommmend alternatives to M2C ? e.g pyopenssl. If you control both ends of the connection then you can do what I did and switch to SSH. The Python Paramiko library implements it really well. (I ended up doing XML-RPC over SSH). Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: pyvm -- faster python
> could You tell us a bit more about Your motivation to create an > alternative C-Python interpreter? I'd also be curious to know if the performance gains would remain once it gets fleshed out with things like closures, long numbers, new style classes and a C library compatibility shim. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Decent Win32All Documentation
"Harlin Seritt" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Does anyone know of any decent documenation on Mark Hammond's win32all > modules? I have tried looking at the documentation .chm file that comes > with it, Python Programming On Win32 (OReilly book) and ActiveState's > documentation and have found them all lacking -- yes I need it broken > down to me. I stringly recommend the platform sdk which includes documentation of *ALL* Windows APIs, sample code etc. It is very easily searchable. win32all is a very thin wrapper over the underlying Windows API. http://www.microsoft.com/msdownload/platformsdk/sdkupdate/ Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: GUI woes
"jeff elkins" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > under debian sid, I installed (via apt-get) the various wxpython stuff > available.: > > libwxgtk2.4-python > libwxgtk2.5.3-python > python-opengl > python-pythoncard > python2.1-opengl > python2.2-opengl > python2.3-opengl > python2.3-pythoncard > pythoncard > pythoncard-doc > pythoncard-tools > wx2.5-examples > wxpython2.5.3 > wxwin2.4-examples You have a mixture of different versions of wxPython in there. > AttributeError: MyFrame instance has no attribute 'Bind' Bind was introduced in wxPython 2.5. Looks like you are running against thr 2.4 library. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.4 killing commercial Windows Python development ?
"Stefan Behnel" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > > Roger Binns schrieb: >> As far as I can tell, they failed at two hurdles. One is that there >> is a new BitPim release every two weeks and they can't really keep up >> with that. (eg it takes around two weeks for packages with a lot of >> attention on Gentoo to become stable and often is a lot longer) > > This is why many open source projects include (possibly outdated) .spec files > directly in their tree. Makes it easy to just adapt > them and run rpmbuild. Similar for Debian package specs. Funnily enough there is an ebuild file in the BitPim source. However it has to be munged for each release since Gentoo requires the ebuild filename to include the version number. I don't bother releasing that file due to the onerous non-automated SourceForge file upload process. (And many of the dependencies aren't in Portage anyway so this would have to be quite a number of ebuilds). > With Python sources it is even easier (most of the time) since you can run > python setup.py bdist_rpm > which spits out a readily baken RPM, ready to be nailed into the system. > Sadly, this doesn't exist for Debian and it doesn't work > for all Python packages (Twisted, that is). The distutils approach is mainly useful for packages and libraries, not for applications. And of course it still has the prerequisites issues mentioned earlier. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.4 killing commercial Windows Python development ?
> Here is the situation I see. I use debian linux systems. Installing all the > dependencies is trivial and if your program has a debian package it would be > a single command. Note that there is *nothing* precluding running using the system Python other than you have to have the dependencies present. In fact that is how all the developers run. The binary/frozen packages are provided as a convenience to the users who just want to use the program and I don't see any need for them to jump through hoops. > The reason I don't like these programs that built the > runtime, static link in a bunch of stuff etc is that it is a pain to upgrade > later. True. People on both Gentoo and Debian have told me they are making proper ebuild/dkpg for BitPim. In both cases nothing came of it which is why I tell people on those systems to just use the rpm. As far as I can tell, they failed at two hurdles. One is that there is a new BitPim release every two weeks and they can't really keep up with that. (eg it takes around two weeks for packages with a lot of attention on Gentoo to become stable and often is a lot longer) The second is dealing with the dependencies. The packager is trying to do BitPim and finds they have to work with others or need assistance to deal with the dependencies. Gentoo is months out of date for wxPython and I have no idea how far behind Debian is. Typically other dependencies aren't even packaged at all, even though they use distutils (ie all it takes is figuring out how to do 'python setup.py install' in a way that keeps the packaging system happy). > If there is a security fix to python 2.4 I know there is ONE copy > installed on the system and that updating it will fix it. If there is a > problem with libpng, libjpeg, kdelibs, zope, apache etc the same is still > true, there is only ONE copy of those items on the system and with a single > command all of them can be updated and fixed. True. Noone forces you to install/run anything. Right now someone on Debian who wants to use BitPim has one of these choices: - Fix Debian's packaging so that it contains all of the dependencies and BitPim itself with the latter being updated every two weeks. This will keep your scenario above on the right track. - Bypass Debian's packaging and install the various dependencies manually and run from "source" - Use alien and the rpm > Keeping track of security updates, feature updates etc for a bunch of > computers with a lot of software from different locations is a royal pain in > the neck. True. And the reality is that various Python packages are backwaters/low priority for the packagers. All it takes is one missing dependency to throw a spanner in the works. And as I told the person who originally wanted to package BitPim for Debian, I will supply all the help and make changes as necessary. Someone from the distros has to step forward to complete it. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.4 killing commercial Windows Python development ?
"Terry Reedy" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > I guess I don't understand some people's determination to not have users > install fully useable Python on their Windows machines. Ok, here is how you install BitPim which contains a frozen Python: - Download and run the setup.exe from www.bitpim.org (The instructions are the equivalent on Linux and Mac) This is how you would do it if a "fully usable" Python had to be put on a machine. - Download and install Python from www.python.org - Download and install wxPython from www.wxpython.org making sure to get the correct platform, Python version, wxPython version and Unicode setting - Download and install pyserial from pyserial.sf.net for your platform - Download and install win32all making sure you get the right Python version (Windows only) - Download and install DSV from sf.net/projects/python-dsv - Download and install APSW from www.rogerbinns.com/apsw.html (Non-Windows users will also have to compile SQLite 3) - Download and install the BitPim code - There are a few other components which non-Windows users typically need and Windows users don't (eg USB library) - Now launch the main Python script to start BitPim The uninstall instructions have the same corresponding lengths. Now for the second part, you could make some arguments: - I shouldn't be using other components in order to reduce dependencies and should instead re-invent the wheel myself. - I could make some sort of installer that did all the non-Python interpretter pieces and it would have to be compatible with anyone else doing the same thing. The first is a waste of my time and effort, and I do the second except I also include the Python interpretter meaning there are no dependencies. > Also, I think it a bit 'anti-social' to hide usage of Python. http://www.bitpim.org/testhelp/3rdparty.htm The reality is that users don't care what language your program was written in, what development methodology you use, how hard it was to write, what editor you use or how your environment enlightens your mind. They do care that what you produce works as expected. In fact if it works really well, they may decide to dig in deeper and try to emulate your language, methodology, procedures, editors in what they do or may contribute to your project if it is open source. That is the point at which Python matters. In all these matters I think it is better to lead by example rather than try to make people aware of things early in order to perform some sort of attempt to gain mindshare. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: PyAsm
"Stefan Behnel" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Meaning: Put the assembler into the doc-string of a function. That has several issues. One is that you can't do string operations with it. Say you wanted some %d, %s etc in the string. If you use a documentation generator (eg epydoc) then the code becomes the API documentation for the function. Finally it bloats binary distributions. For example BitPim is 7-10MB binary distribution, full compressed with all doc strings removed. Including doc strings adds another 3MB to the compressed binary size! Instead I would suggest looking at the compile/eval/exec builtins in Python for inspiration. You can give a string to compile and it gives you something you can execute later in varying contexts. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: FTPLIB & FTPS or SFTP?
"Peter A. Schott" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > have a handful of partners who use FTPS or SFTP and I need to pull/push files > to/from them. For SFTP, run don't walk, over to http://www.lag.net/paramiko/ Paramiko is a pure Python(*) implementation of SSH and all the sub-protocols such as SFTP. Works on all platforms, very responsive author, works reliably and LGPL license. (*) It does require PyCrypto which is a native code package available for all platforms. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: [ANN] iCalendar package 0.9
"Max M" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > http://www.mxm.dk/products/public/ical/ > > Any feedback would be welcome. How well do you cope with the crud that real programs generate? Does it work with the different dialects uses out there? Can it at least identify them? The reason why I ask is this would be useful for BitPim. When I wrote the code for doing vCards, I soon found that there wasn't actually a single program out there on Windows, Linux or Mac that actually generated 100% standards conformant vCards. They generally complied with the spirit, but screwed up character encoding, misspelled fields names, didn't do the right thing when commas and semi-colons were present in values etc. I assume the thing happens with ical. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: there's a socket.sendall(), so why no socket.recvall()?
there's a socket.sendall(), so why no socket.recvall()? BTW socket.sendall() doesn't actually work for large amounts of data on Windows 2000 and probably other versions of Windows as well. Eg if you supply a 1MB buffer then you get an exception based on some internal Windows error code. I haven't experimented on Unix yet to see if it has the same issue. The workaround is to write a wrapper that really does send everything. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: PyChecker messages
> runner.py:878: Function (main) has too many lines (201) > > What does this mean? Cannot functions be large? Or is it simply an advice that > functions should be small and simple? It is advice. > runner.py:200: Function (detectMimeType) has too many returns (11) > > The function is simply a long "else-if" clause, branching out to different > return statements. What's wrong? It's simply a "probably ugly code" advice? That is also advice. Generally you use a dict of functions, or some other structure to lookup what you want to do. > _must_ take two arguments; is there any way that I can make 'frame' go away? Yes, you can name it just _ (underscore). There are a few other names like that that are ignored. (Check the doc). > Also, another newbie question: How does one make a string stretch over several > lines in the source code? Is this the proper way? > > print "asda asda asda asda asda asda " \ > "asda asda asda asda asda asda " \ > "asda asda asda asda asda asda" Depends what you are trying to achieve. Look at triple quoting as well. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Recent infoworld column
"Peter Hansen" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Dwarf Electrician wrote: >> from a long time listener... >> >> http://www.infoworld.com/article/04/12/30/01OPstrategic_1.html > > Kudos for Roger Binns! It is a very nice article :-) BitPim like many other projects is an effort by several people, but what John did was entirely my code. And of course that code wouldn't be possible without the various components I had available to me which including Python and wxPython as well as several others. http://bitpim.org/testhelp/credits.htm http://bitpim.org/testhelp/3rdparty.htm You may also find a talk I gave at baypiggies in July 2004 of interest. http://bitpim.org/papers/baypiggies/ It covers the various issues in doing a "real world" Python application, including packaging them up so they are indistinguishable from native applications, accessing serial ports, USB and SWIG, threading, the GUIs available and why I picked wxPython, Outlook and Evolution integration, dealing with an undocumented binary protocol, user and programmer documentation, secure remote access etc. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: PHP vs. Python
"Stephen" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] >I like the idea of being able to port specific sections to C ... Python > seems more flexible than PHP ... scalable. If you want portions of your code in C, then wrap them with Swig. That way they can be available in any number of languages including Python, PHP, Java and Perl. http://www.swig.org/compare.html Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: PHP vs. Python
"Eric Pederson" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > My beloved Python-oriented webhost doesn't currently support Mod-Python You can always do what I did. I wrote the backend of my app in Python and run it as an XML-RPC server. I did the front end in PHP using the Smarty template tool. (The actual templates themselves were stored in the Python server and grabbed via XML-RPC). Effectively PHP/Smarty were formatting XML-RPC results, delivered as Python dicts which turn into Smarty arrays (and it all works fine with nested lists and dicts). That way I got the best of both worlds, didn't have to get mod-python installed, and *my opinion* is that Smarty is the nicest template tool I have tried in Python or PHP. Roger -- http://mail.python.org/mailman/listinfo/python-list
Source cross reference + colourizer (PyXR is no more?)
I have been a happy user of PyXR which colourizes source to HTML and also cross references it. Here is an example of the output: http://bitpim.org/pyxr/c/projects/bitpim/analyser.py.html Unfortunately the author and his site appears to have gone AWOL for quite a while. It used to be: http://www.cathoderaymission.net/~logistix/python/pyxr.html Does anyone know what happened to the author? Are there any maintained alternatives to PyXR? Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Boo who? (was Re: newbie question)
"Fredrik Lundh" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Terry Reedy wrote: > >> This I again agree with. I understand that Prothon is also a *different* >> though Python inspired language. Also that it is still >> under development. > > http://www.prothon.org/ > >"All work on Prothon has been halted" > > (the site is pointing to a new project, Pycs, whose site is pointing to a new > project, Spry, "the first dynamic language to have all the best capabilities > of > Python, Prothon, and C# (actually C-Omega) in one language", on which > "work has rarely begun") That work died due to "a crisis of faith": http://mylist.net/archives/spry-dev/2004-November/72.html Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: why no python setup.py uninstall?
<[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > I think a little database (maybe in xml?) of installed files/modules > and their locations would be useful, perhaps even for a future > automatic download/installation/dependency-tracking thingmabob that > still regretably still doesn't exist as parth of the stdlib in > Pythonia. Actually it does, indirectly Instead of doing 'python setup.py install', change the last parameter to bdist. You can add --format wininst, --format rpm and various other choices to make a package native to your platform. You can then use your platform tools to install and uninstall the files. Or you can use the zip/tar formats and keep track of the files by whatever means you want to. http://docs.python.org/dist/built-dist.html Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Fun with Outlook and MAPI
"Larry Bates" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > At the risk of beating a dead horse, but you really should consider > using SMTP instead if you are really going to be sending a lot > of messages. The problem is that doesn't work in more complicated configurations such as when authentication and/or SSL have to happen, not to mention the issue of configuring servers and ports, that users have already configured in their mail clients. Additionally when you use MAPI, messages that you send also end up in your sent items folder. (I also have not had any issue sending lots of messages using MAPI). Ultimately the utility of vanilla of pure SMTP will depend on customer requirements and how simple the configuration is. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Fun with Outlook and MAPI
"Chris" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > My only problem is that when I call Resolve() and Send(), I get confirmation > dialogs. I will be sending out quite a few e-mails > at a time, and would rather not have the user have to click yes for every > single one. Here is an simple workaround: http://www.contextmagic.com/express-clickyes/ If you have win32all installed, you can even use the programming API they present to turn on and off the clickyes functionality. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: PajamaScript
"Jerome Chan" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] >I wrote something called PajamaScript. Basically, it parses a text > file and looks for tags. Then it calls python to handle the > scripting. Why learn another language when you already know Python? Why write another templating tool, when there are so many already (other than as a valuable learning exercise). Check out the Python Cookbook: http://aspn.activestate.com/ASPN/Python/Cookbook/ It has several. Here are the best: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52217 http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/162292 Note that they also deal with the area you will have an issue in which is how to add conditional statements. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: PySQLLite Speed
"Kevin" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Then when it starts to > write the Database, the PC Util drops to 1-2% and it > takes forever. I'm not PC related preformance > barriers that I'm aware of. Your hard disk. See the synchronous information in the pragmas: http://www.sqlite.org/pragma.html See also: http://www.sqlite.org/cvstrac/wiki?p=PerformanceConsiderations http://www.sqlite.org/cvstrac/wiki?p=PerformanceTuning Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: What to use for installation?
"Eugene Morozov" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > But I want to know other opinions before going with scons. What is the best > tool for installing python applications? The tools you mention are actually more applicable to Python packages in that they install the code into an existing Python install on the machine. For applications most people want standalone programs that the user doesn't know or care is written in Python and doesn't have any dependencies on things already installed on their system. In that case you can use a Freezing tool which will package up an executable stub, your code, the interpretter dll/so and other binary/source modules you may use (eg GTK in your case). This will give you a single directory with no dependcies that will run your program. Then you can use an installer that is platform specific. For example on Windows you'll want the program to install from a setup.exe and be in the Add/Remove programs list in the Contol Panel. On Linux you'll want it added to KDE/Gnome user menus. For freezing tools, there are: - cx-Freeze (Windows, Linux, Mac?) - py2exe (Windows) - McMillan Installer (Windows, Linux, Mac?) - py2app (Mac) - bundlebuilder (Mac) For installers, there are: - InnoSetup (Windows) - NSIS (Windows) - rpm/dpkg (Linux) - dimg tools (Mac) - EPM (Linux/Unix) IMHO the best of breed are Windows: py2exe/InnoSetup Linux:cx-Freeze/rpm Mac: py2app/dimg Other people use other combinations and have their favourites. You can see some slides from a talk I did a while back that goes into a little more detail, as well as giving an idea of how much "code"/templates were written for the bitpim project to use a freezer/installer on Windows, Linux and Mac. http://bitpim.org/papers/baypiggies/ (See slides 18 thru 22). Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: wxPython to build an HTML analyser/displayer?
"Luke Skywalker" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Does wxWidgets offer an HTML displayer widget, Yes. In general it is highly recommended to download wxPython and the associated demo app. The demo app shows every single widget so you get an idea of what is available, what they look like and you can examine the demo code to see how they are used. > and is it good enough? Unlikely. It is a simple HTML viewer and doesn't support stylesheets amongst other things. (It does support adding your own tags, widgets and virtual filesystems though). More details are at http://www.lpthe.jussieu.fr/~zeitlin/wxWindows/docs/wxwin_wxhtml.html (The "Tags supported by wxHTML 75% of the way down will be useful information for you) However the good news is you have two alternatives. One is that on Windows you can use the IE HTML display widget directly. See the demo. If you need to be cross platform then wxMozilla is worth looking at (which is apparently active despite appearing dormant). Start at http://wxmozilla.sourceforge.net/pmwiki/pmwiki.php/Main/Documentation Roger -- http://mail.python.org/mailman/listinfo/python-list
