Re: If Scheme is so good why MIT drops it?
In article urr9m.6558$ze1.5...@news-server.bigpond.net.au, Neil Hodgson nyamatongwe+thun...@gmail.com wrote: milanj: and all of them use native threads (python still use green threads ?) Python uses native threads. But then it adds the global interpreter lock, which completely undermines the utility of native threads. So yes, it uses native threads, but it does not actually realize the benefits of that use. rg -- http://mail.python.org/mailman/listinfo/python-list
Yet another unicode WTF
Python 2.6.2 on OS X 10.5.7: [...@mickey:~]$ echo $LANG en_US.UTF-8 [...@mickey:~]$ cat frob.py #!/usr/bin/env python print u'\u03BB' [...@mickey:~]$ ./frob.py ª [...@mickey:~]$ ./frob.py foo Traceback (most recent call last): File ./frob.py, line 2, in module print u'\u03BB' UnicodeEncodeError: 'ascii' codec can't encode character u'\u03bb' in position 0: ordinal not in range(128) (That's supposed to be a small greek lambda, but I'm using a brain-damaged news reader that won't let me set the character encoding. It shows up correctly in my terminal.) According to what I thought I knew about unix (and I had fancied myself a bit of an expert until just now) this is impossible. Python is obviously picking up a different default encoding when its output is being piped to a file, but I always thought one of the fundamental invariants of unix processes was that there's no way for a process to know what's on the other end of its stdout. Clues appreciated. Thanks. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Yet another unicode WTF
In article h09ten$5q...@lust.ihug.co.nz, Lawrence D'Oliveiro l...@geek-central.gen.new_zealand wrote: In message rnospamon-e7e08b.18181804062...@news.gha.chartermi.net, Ron Garret wrote: Python 2.6.2 on OS X 10.5.7: Same result, Python 2.6.1-3 on Debian Unstable. My $LANG is en_NZ.UTF-8. ... I always thought one of the fundamental invariants of unix processes was that there's no way for a process to know what's on the other end of its stdout. Well, there have long been functions like isatty(3). That's probably what's involved here. Oh. Right. Duh. I am having an unbelievably bad day involving lawyers. (And not language lawyers, real ones.) Found the answer here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=415968 rg -- http://mail.python.org/mailman/listinfo/python-list
distutils extension configuration problem
I'm trying to build PyObjC on an Intel Mac running OS X 10.5.7. The build is breaking because distutils seems to want to build extension modules as universal binaries, but some of the libraries it depends on are built for intel-only, i.e.: [...@mickey:~/Desktop/pyobjc-framework-ScreenSaver-2.2b2]$ python2.6 setup.py build /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils /dist.py:266: UserWarning: Unknown distribution option: 'options' warnings.warn(msg) running build running build_py running build_ext building 'ScreenSaver._inlines' extension gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g -bundle -undefined dynamic_lookup build/temp.macosx-10.3-i386-2.6/Modules/_ScreenSaver_inlines.o -o build/lib.macosx-10.3-i386-2.6/ScreenSaver/_inlines.so -framework ScreenSaver ld: in /Developer/SDKs/MacOSX10.4u.sdk/usr/local/lib/libTIFF.dylib, file is not of required architecture for architecture ppc collect2: ld returned 1 exit status lipo: can't open input file: /var/folders/nT/nTiypn-v2RatkU+BYncrKU+++TI/-Tmp-//ccMFYRkt.out (No such file or directory) error: command 'gcc' failed with exit status 1 [...@mickey:~/Desktop/pyobjc-framework-ScreenSaver-2.2b2]$ file build/temp.macosx-10.3-i386-2.6/Modules/_ScreenSaver_inlines.o build/temp.macosx-10.3-i386-2.6/Modules/_ScreenSaver_inlines.o: Mach-O universal binary with 2 architectures build/temp.macosx-10.3-i386-2.6/Modules/_ScreenSaver_inlines.o (for architecture ppc): Mach-O object ppc build/temp.macosx-10.3-i386-2.6/Modules/_ScreenSaver_inlines.o (for architecture i386): Mach-O object i386 [...@mickey:~/Desktop/pyobjc-framework-ScreenSaver-2.2b2]$ file /usr/local/lib/libtiff.dylib /usr/local/lib/libtiff.dylib: Mach-O dynamically linked shared library i386 How do I get distutils to stop trying to build extensions as universal binaries? Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Re: To unicode or not to unicode
In article mailman.373.1235153296.11746.python-l...@python.org, MRAB goo...@mrabarnett.plus.com wrote: Thorsten Kampe wrote: * Ron Garret (Thu, 19 Feb 2009 18:57:13 -0800) I'm writing a little wiki that I call µWiki. That's a lowercase Greek mu at the beginning (it's pronounced micro-wiki). No, it's not. I suggest you start your Unicode adventure by configuring your newsreader. It looked like mu to me, but you're correct: it's MICRO SIGN, not GREEK SMALL LETTER MU. Heh, I didn't know that those two things were distinct. Learn something new every day. rg -- http://mail.python.org/mailman/listinfo/python-list
What encoding does u'...' syntax use?
I would have thought that the answer would be: the default encoding
(duh!) But empirically this appears not to be the case:
unicode('\xb5')
Traceback (most recent call last):
File stdin, line 1, in module
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 0:
ordinal not in range(128)
u'\xb5'
u'\xb5'
print u'\xb5'
µ
(That last character shows up as a micron sign despite the fact that my
default encoding is ascii, so it seems to me that that unicode string
must somehow have picked up a latin-1 encoding.)
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: To unicode or not to unicode
In article 499f0cf0.8070...@v.loewis.de, Martin v. Löwis mar...@v.loewis.de wrote: MRAB wrote: Thorsten Kampe wrote: * Ron Garret (Thu, 19 Feb 2009 18:57:13 -0800) I'm writing a little wiki that I call µWiki. That's a lowercase Greek mu at the beginning (it's pronounced micro-wiki). No, it's not. I suggest you start your Unicode adventure by configuring your newsreader. It looked like mu to me, but you're correct: it's MICRO SIGN, not GREEK SMALL LETTER MU. I don't think that was the complaint. Instead, the complaint was that the OP's original message did not have a Content-type header, I'm the OP. I'm using MT-Newswatcher 3.5.1. I thought I had it configured properly, but I guess I didn't. Under Preferences-Languages-Send Messages with Encoding I had selected latin-1. I didn't know I also needed to have MIME turned on for that to work. I've turned it on now. Is this better? This should be a micro sign: µ rg -- http://mail.python.org/mailman/listinfo/python-list
Re: What encoding does u'...' syntax use?
In article 499f18bd$0$31879$9b4e6...@newsspool3.arcor-online.net,
Stefan Behnel stefan...@behnel.de wrote:
Ron Garret wrote:
I would have thought that the answer would be: the default encoding
(duh!) But empirically this appears not to be the case:
unicode('\xb5')
Traceback (most recent call last):
File stdin, line 1, in module
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 0:
ordinal not in range(128)
u'\xb5'
u'\xb5'
print u'\xb5'
µ
(That last character shows up as a micron sign despite the fact that my
default encoding is ascii, so it seems to me that that unicode string
must somehow have picked up a latin-1 encoding.)
You are mixing up console output and internal data representation. What you
see in the last line is what the Python interpreter makes of your unicode
string when passing it into stdout, which in your case seems to use a
latin-1 encoding (check your environment settings for that).
BTW, Unicode is not an encoding. Wikipedia will tell you more.
Yes, I know that. But every concrete representation of a unicode string
has to have an encoding associated with it, including unicode strings
produced by the Python parser when it parses the ascii string u'\xb5'
My question is: what is that encoding? It can't be ascii. So what is
it?
Put this another way: I would have thought that when the Python parser
parses u'\xb5' it would produce the same result as calling
unicode('\xb5'), but it doesn't. Instead it seems to produce the same
result as calling unicode('\xb5', 'latin-1'). But my default encoding
is not latin-1, it's ascii. So where is the Python parser getting its
encoding from? Why does parsing u'\xb5' not produce the same error as
calling unicode('\xb5')?
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: What encoding does u'...' syntax use?
In article 499f3a8f.9010...@v.loewis.de, Martin v. Löwis mar...@v.loewis.de wrote: u'\xb5' u'\xb5' print u'\xb5' ? Unicode literals are *in the source file*, which can only have one encoding (for a given source file). (That last character shows up as a micron sign despite the fact that my default encoding is ascii, so it seems to me that that unicode string must somehow have picked up a latin-1 encoding.) I think latin-1 was the default without a coding cookie line. (May be uft-8 in 3.0). It is, but that's irrelevant for the example. In the source u'\xb5' all characters are ASCII (i.e. all of letter u, single quote, backslash, letter x, letter b, digit 5). As a consequence, this source text has the same meaning in all supported source encodings (as source encodings must be ASCII supersets). The Unicode literal shown here does not get its interpretation from Latin-1. Instead, it directly gets its interpretation from the Unicode coded character set. The string is a short-hand for u'\u00b5' and this denotes character U+00B5 (just as u'\u20ac denotes U+20AC; the same holds for any other u'\u'). HTH, Martin Ah, that makes sense. Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: What encoding does u'...' syntax use?
In article 499f397c.7030...@v.loewis.de,
Martin v. Löwis mar...@v.loewis.de wrote:
Yes, I know that. But every concrete representation of a unicode string
has to have an encoding associated with it, including unicode strings
produced by the Python parser when it parses the ascii string u'\xb5'
My question is: what is that encoding?
The internal representation is either UTF-16, or UTF-32; which one is
a compile-time choice (i.e. when the Python interpreter is built).
Put this another way: I would have thought that when the Python parser
parses u'\xb5' it would produce the same result as calling
unicode('\xb5'), but it doesn't.
Right. In the former case, \xb5 denotes a Unicode character, namely
U+00B5, MICRO SIGN. It is the same as u\u00b5, and still the same
as u\N{MICRO SIGN}. By the same, I mean the very same.
OTOH, unicode('\xb5') is something entirely different. '\xb5' is a
byte string with length 1, with a single byte with the numeric
value 0xb5, or 181. It does not, per se, denote any specific character.
It only gets a character meaning when you try to decode it to unicode,
which you do with unicode('\xb5'). This is short for
unicode('\xb5', sys.getdefaultencoding())
and sys.getdefaultencoding() is (or should be) ascii. Now, in
ASCII, byte 0xb5 does not have a meaning (i.e. it does not denote
a character at all), hence you get a UnicodeError.
Instead it seems to produce the same
result as calling unicode('\xb5', 'latin-1').
Sure. However, this is only by coincidence, because latin-1 has the same
code points as Unicode (for 0..255).
But my default encoding
is not latin-1, it's ascii. So where is the Python parser getting its
encoding from? Why does parsing u'\xb5' not produce the same error as
calling unicode('\xb5')?
Because \xb5 *directly* refers to character U+00b5, with no
byte-oriented encoding in-between.
Regards,
Martin
OK, I think I get it now. Thanks!
rg
--
http://mail.python.org/mailman/listinfo/python-list
Regular expression bug?
I'm trying to split a CamelCase string into its constituent components.
This kind of works:
re.split('[a-z][A-Z]', 'fooBarBaz')
['fo', 'a', 'az']
but it consumes the boundary characters. To fix this I tried using
lookahead and lookbehind patterns instead, but it doesn't work:
re.split('((?=[a-z])(?=[A-Z]))', 'fooBarBaz')
['fooBarBaz']
However, it does seem to work with findall:
re.findall('(?=[a-z])(?=[A-Z])', 'fooBarBaz')
['', '']
So the regular expression seems to be doing the Right Thing. Is this a
bug in re.split, or am I missing something?
(BTW, I tried looking at the source code for the re module, but I could
not find the relevant code. re.split calls sre_compile.compile().split,
but the string 'split' does not appear in sre_compile.py. So where does
this method come from?)
I'm using Python2.5.
Thanks,
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: Regular expression bug?
In article mailman.281.1235073821.11746.python-l...@python.org,
MRAB goo...@mrabarnett.plus.com wrote:
Ron Garret wrote:
I'm trying to split a CamelCase string into its constituent components.
This kind of works:
re.split('[a-z][A-Z]', 'fooBarBaz')
['fo', 'a', 'az']
but it consumes the boundary characters. To fix this I tried using
lookahead and lookbehind patterns instead, but it doesn't work:
re.split('((?=[a-z])(?=[A-Z]))', 'fooBarBaz')
['fooBarBaz']
However, it does seem to work with findall:
re.findall('(?=[a-z])(?=[A-Z])', 'fooBarBaz')
['', '']
So the regular expression seems to be doing the Right Thing. Is this a
bug in re.split, or am I missing something?
(BTW, I tried looking at the source code for the re module, but I could
not find the relevant code. re.split calls sre_compile.compile().split,
but the string 'split' does not appear in sre_compile.py. So where does
this method come from?)
I'm using Python2.5.
I, amongst others, think it's a bug (or 'misfeature'); Guido thinks it
might be intentional, but changing it could break some existing code.
That seems unlikely. It would only break where people had code invoking
re.split on empty matches, which at the moment is essentially a no-op.
It's hard to imagine there's a lot of code like that around. What would
be the point?
You could do this instead:
re.sub('(?=[a-z])(?=[A-Z])', '@', 'fooBarBaz').split('@')
['foo', 'Bar', 'Baz']
Blech! ;-) But thanks for the suggestion.
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: Regular expression bug?
In article gnkdal$bcq$0...@news.t-online.com, Peter Otten __pete...@web.de wrote: Ron Garret wrote: I'm trying to split a CamelCase string into its constituent components. How about re.compile([A-Za-z][a-z]*).findall(fooBarBaz) ['foo', 'Bar', 'Baz'] That's very clever. Thanks! (BTW, I tried looking at the source code for the re module, but I could not find the relevant code. re.split calls sre_compile.compile().split, but the string 'split' does not appear in sre_compile.py. So where does this method come from?) It's coded in C. The source is Modules/sremodule.c. Ah. Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Regular expression bug?
In article mailman.277.1235073073.11746.python-l...@python.org,
andrew cooke and...@acooke.org wrote:
i wonder what fraction of people posting with bug? in their titles here
actually find bugs?
IMHO it ought to be an invariant that len(r.split(s)) should always be
one more than len(r.findall(s)).
anyway, how about:
re.findall('[A-Z]?[a-z]*', 'fooBarBaz')
or
re.findall('([A-Z][a-z]*|[a-z]+)', 'fooBarBaz')
That will do it. Thanks!
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: Regular expression bug?
In article mailman.273.1235071607.11746.python-l...@python.org,
Albert Hopkins mar...@letterboxes.org wrote:
On Thu, 2009-02-19 at 10:55 -0800, Ron Garret wrote:
I'm trying to split a CamelCase string into its constituent components.
This kind of works:
re.split('[a-z][A-Z]', 'fooBarBaz')
['fo', 'a', 'az']
but it consumes the boundary characters. To fix this I tried using
lookahead and lookbehind patterns instead, but it doesn't work:
That's how re.split works, same as str.split...
I think one could make the argument that 'foo'.split('') ought to return
['f','o','o']
re.split('((?=[a-z])(?=[A-Z]))', 'fooBarBaz')
['fooBarBaz']
However, it does seem to work with findall:
re.findall('(?=[a-z])(?=[A-Z])', 'fooBarBaz')
['', '']
Wow!
To tell you the truth, I can't even read that...
It's a regexp. Of course you can't read it. ;-)
rg
--
http://mail.python.org/mailman/listinfo/python-list
To unicode or not to unicode
I'm writing a little wiki that I call µWiki. That's a lowercase Greek mu at the beginning (it's pronounced micro-wiki). It's working, except that I can't actually enter the name of the wiki into the wiki itself because the default unicode encoding on my Python installation is ascii. So I'm trying to decide on a course of action. There seem to be three possibilities: 1. Change the code to properly support unicode. Preliminary investigations indicate that this is going to be a colossal pain in the ass. 2. Change the default encoding on my Python installation to be latin-1 or UTF8. The disadvantage to this is that no one else will be able to run my code without making the same change to their installation, since you can't change default encodings once Python has started. 3. Punt and spell it 'uwiki' instead. I'm feeling indecisive so I thought I'd ask other people's opinion. What should I do? rg -- http://mail.python.org/mailman/listinfo/python-list
Re: WebError documentation?
In article mailman.9037.1233981452.3487.python-l...@python.org, Chris Rebert c...@rebertia.com wrote: On Thu, Feb 5, 2009 at 12:52 PM, Ron Garret rnospa...@flownet.com wrote: Is there any? Where is it? Extensive Googling has proven fruitless. It's not a standard Python exception. A third-party library you're using must be raising it. Check the exception traceback. I see I did not make myself clear. I meant the Python software package called weberror, not a generic web error. http://pypi.python.org/pypi/WebError rg -- http://mail.python.org/mailman/listinfo/python-list
Re: WebError documentation?
In article 498de947$0$24412$426a7...@news.free.fr, Bruno Desthuilliers bdesth.quelquech...@free.quelquepart.fr wrote: Ron Garret a écrit : In article mailman.9037.1233981452.3487.python-l...@python.org, Chris Rebert c...@rebertia.com wrote: On Thu, Feb 5, 2009 at 12:52 PM, Ron Garret rnospa...@flownet.com wrote: Is there any? Where is it? Extensive Googling has proven fruitless. It's not a standard Python exception. A third-party library you're using must be raising it. Check the exception traceback. I see I did not make myself clear. Now *that* is an understatement... I meant the Python software package called weberror, not a generic web error. http://pypi.python.org/pypi/WebError Google is your friend: http://bel-epa.com/docs/thirdparty/weberror/ Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
WebError documentation?
Is there any? Where is it? Extensive Googling has proven fruitless. Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Extreme Yaro weirdness
I'm running the following WSGI app under Yaro:
def error(req):
try:
req.non_existent_key
except:
try:
return cgitb.html(sys.exc_info())
except:
return 'foo'
The result of running this is 'foo'. In other words, the reference to
the non-existent key generates an exception as expected, but then the
generation of the traceback generates ANOTHER exception, which I would
not have expected. If I add another TRY/EXCEPT block to capture this
exception, the result is attached at the bottom of this message.
It gets better.
I get the same result whether I run under Apache/mod_wsgi or standalone
using wsgi_ref. But if I add the following code to try to capture a
Yaro request object so I can noodle around with it:
def error(req):
global req1
req1=req
try: ...
it makes the problem go away! But it ONLY makes the problem go away
when running under wsgi_ref, not under Apache!
The problem seems to be specific to Yaro. If I make an instance of an
object and try to dereference a non-existent attribute, the initial
traceback gets generated with no problem. Also, the Yaro code for
Request.__getattr__ ultimately calls req.__dict__[attr]. If I make this
call directly, the problem also goes away.
I am quite baffled by this. Any suggestions on how to debug this would
be much appreciated.
Thanks,
rg
type 'exceptions.KeyError'
Python 2.5: /usr/local/bin/python
Mon Feb 2 00:54:33 2009
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/www/sites/mcia.cc/files/wsgi/test.wsgi in error(req=yaro.Request
object at 0x2b94a690)
212 except:
213 try:
214 return cgitb.text(sys.exc_info())
215 except:
216 try:
global cgitb = module 'cgitb' from '/usr/lib/python2.5/cgitb.pyc'
cgitb.text = function text at 0x2b0adc80
global sys = module 'sys' (built-in)
sys.exc_info = built-in function exc_info
/www/sites/cgitb.py in text((etype=type 'exceptions.KeyError',
evalue=KeyError('non_existent_key',), etb=traceback object at
0x2b960368), context=5)
215 try: return linecache.getline(file, lnum[0])
216 finally: lnum[0] += 1
217 vars = scanvars(reader, frame, locals)
218
219 rows = [' %s %s' % (file, call)]
builtinvars = built-in function vars
global scanvars = function scanvars at 0x2b0adb90
reader = function reader at 0x2bbbea28
frame = frame object at 0xba49e0
locals = {'req': yaro.Request object at 0x2b94a690}
/www/sites/cgitb.py in scanvars(reader=function reader at
0x2bbbea28, frame=frame object at 0xba49e0, locals={'req':
yaro.Request object at 0x2b94a690})
82 if lasttoken == '.':
83 if parent is not __UNDEF__:
84 value = getattr(parent, token, __UNDEF__)
85 vars.append((prefix + token, prefix, value))
86 else:
value = yaro.Request object at 0x2b94a690
builtingetattr = built-in function getattr
parent = yaro.Request object at 0x2b94a690
token = 'non_existent_key'
__UNDEF__ undefined
/www/sites/mcia.cc/files/wsgi/yaro.py in __getattr__(self=yaro.Request
object at 0x2b94a690, attr='non_existent_key')
215 elif attr == 'cookie' and not 'cookie' in self.__dict__:
216 self._load_cookie()
217 return self.__dict__[attr]
218
219 def _parse_query(self):
self = yaro.Request object at 0x2b94a690
self.__dict__ = {'_start_response': built-in method start_response of
mod_wsgi.Adapter object at 0x2ab218a0, 'content_length': '',
'content_type': '', 'environ': {'DOCUMENT_ROOT':
'/www/sites/mcia.cc/html', 'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT':
'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plai
n;q=0.8,image/png,*/*;q=0.5', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate',
'HTTP_ACCEPT_LANGUAGE': 'en-us', 'HTTP_CACHE_CONTROL': 'max-age=0',
'HTTP_CONNECTION': 'keep-alive', 'HTTP_COOKIE': 'sid=55UGS64T',
'HTTP_HOST': 'mcia.cc', 'HTTP_REFERER': 'http://mcia.cc/wtest/menu',
...}, 'exc_info': None, 'extra_props': None, 'method': 'GET', 'query':
{}, 'res': yaro.Response object at 0x2b94a710,
'start_response_called': False, ...}
attr = 'non_existent_key'
type 'exceptions.KeyError': 'non_existent_key'
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File /www/sites/mcia.cc/files/wsgi/test.wsgi, line 214, in error
return cgitb.text(sys.exc_info())
File cgitb.py, line 217, in text
vars = scanvars(reader, frame, locals)
File cgitb.py, line 84, in scanvars
value = getattr(parent, token, __UNDEF__)
File /www/sites/mcia.cc/files/wsgi/yaro.py, line 217, in __getattr__
return self.__dict__[attr]
KeyError: 'non_existent_key'
--
http://mail.python.org/mailman/listinfo/python-list
Re: Extreme Yaro weirdness
In article mailman.8612.1233589195.3487.python-l...@python.org, Gabriel Genellina gagsl-...@yahoo.com.ar wrote: En Mon, 02 Feb 2009 06:59:16 -0200, Ron Garret rnospa...@flownet.com escribió: I'm running the following WSGI app under Yaro: def error(req): try: req.non_existent_key except: try: return cgitb.html(sys.exc_info()) except: return 'foo' The result of running this is 'foo'. In other words, the reference to the non-existent key generates an exception as expected, but then the generation of the traceback generates ANOTHER exception, which I would not have expected. If I add another TRY/EXCEPT block to capture this exception, the result is attached at the bottom of this message. Unqualified excepts are evil -- see this recent post (mine) http://groups.google.com/group/comp.lang.python/msg/05e822f694b6421c But in this case you hit a known bug in cgitb - see http://bugs.python.org/issue4643 for a solution. Aha! Thank you! That explains everything, except this: why does the problem go away when I run under wsgiref and capture the request in a global variable? Feel free to treat that as a rhetorical question :) rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Sloooooowwwww WSGI restart
In article 146f6796-37b5-4220-bdb1-5119cb3ac...@z6g2000pre.googlegroups.com, Graham Dumpleton graham.dumple...@gmail.com wrote: On Jan 30, 9:53 am, Ron Garret rnospa...@flownet.com wrote: In article 498171a5$0$3681$426a7...@news.free.fr, Bruno Desthuilliers bruno.42.desthuilli...@websiteburo.invalid wrote: Ron Garret a écrit : In article mailman.8228.1233179089.3487.python-l...@python.org, Aleksandar Radulovic a...@a13x.net wrote: (snip) Secondly, why are you restarting apache after code changes? In normal circumstances, you shouldn't have to do that. I thought (and experiment confirms) that only the main WSGI app file gets reloaded automatically when it changes, not the libraries. Depends on how you configure mod_wsgi. Read the part about Process Reloading Mechanism here: http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode I'm running Debian Etch, which means I have an older version of mod_wsgi, which means I don't have the process reloading mechanism. Back port available at: http://packages.debian.org/etch-backports/libapache2-mod-wsgi Graham Cool! Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: More mod_wsgi weirdness: process restarts on redirect
In article 63cf7deb-f15c-4259-aa24-1b8da8468...@r41g2000prr.googlegroups.com, Graham Dumpleton graham.dumple...@gmail.com wrote: On Jan 30, 11:01 am, Ron Garret rnospa...@flownet.com wrote: In article mailman.8321.1233272610.3487.python-l...@python.org, Joshua Kugler jos...@joshuakugler.com wrote: Ron Garret wrote: My question is: is this supposed to be happening? Or is this an indication that something is wrong, and if so, what? You are probably just hitting a different instance of Apache, thus the different process ID. Yep, that's what it turned out to be. I thought I had a WSGIDaemonProcess processes=1 directive in my config, but I had it in the wrong place (a different vhost) so it wasn't actually active. http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading But that leaves me wondering why the redirect would reliably trigger switching processes. The reason I thought that I had the correct configuration and only had one process is that when I reloaded the non-redirected page I *always* got the same process ID. How doesmod_wsgidecide which process (and which thread for that matter) to use? Details on process/threading in mod_wsgi available at: http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading When using WSGIDaemonProcess directive, if you want a single process it is better to allow it to default to a single process and not have 'processes=1'. As soon as you say 'processes=1' it will trigger wsgi.multiprocess to be True rather than default of False. This may sound counter intuitive, but is a little back door to allow wsgi.multiprocess to be set to True somehow when distributing an application across a cluster of machines where it does need to be True even if each machine only has a single process for that application. Tthat wsgi.multiprocess is True will not usually matter unless you are trying to use debugging middleware that require that there only be a single process. As to why you were getting a different process, because you were actually running in embedded mode due to WSGIDaemonProcess/ WSGIProcessGroup being in wrong context, then what process was used was really up to Apache and how it works. Specifically it can have multiple processes that can listen on the HTTP port (80). Because only one should be listening at a time it uses a cross process mutex lock to mediate access. When a process handles a request, it gives up the lock. If using worker MPM then another thread in same process may get lock, or for either worker MPM or prefork MPM, then another process could get it. Which actually gets it is a bit indeterminate as simply depends on which process the operating system lets have the lock next. So, there is no strict rule one can say as to who would get it next. Graham Graham Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Sloooooowwwww WSGI restart
In article 498171a5$0$3681$426a7...@news.free.fr, Bruno Desthuilliers bruno.42.desthuilli...@websiteburo.invalid wrote: Ron Garret a écrit : In article mailman.8228.1233179089.3487.python-l...@python.org, Aleksandar Radulovic a...@a13x.net wrote: (snip) Secondly, why are you restarting apache after code changes? In normal circumstances, you shouldn't have to do that. I thought (and experiment confirms) that only the main WSGI app file gets reloaded automatically when it changes, not the libraries. Depends on how you configure mod_wsgi. Read the part about Process Reloading Mechanism here: http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode I'm running Debian Etch, which means I have an older version of mod_wsgi, which means I don't have the process reloading mechanism. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Sloooooowwwww WSGI restart
In article 498170d4$0$23718$426a7...@news.free.fr, Bruno Desthuilliers bruno.42.desthuilli...@websiteburo.invalid wrote: Ron Garret a écrit : I'm running a WSGI app under apache/mod_wsgi and I've noticed that whenever I restart the server after making a code change it takes a very long time (like a minute) before the script is active again. In other words, I do an apachectl restart, reload the page in my browser, and one minute later it finally comes up. During this time CPU usage is essentially zero. Loading all the code manually into a python interpreter is virtually instantaneous, and all subsequence interactions with the app are very fast. Does anyone have any ideas what might be going on or how to debug this? Restarting apache (with or without mod_wsgi) can by itself take some time. Now, if you're running mod_wsgi in daemon mode, you _don't_ have to restart apache to reload your code - just touch the wsgi script file and you'll be done, ie with: WSGIProcessGroup myproject.tld WSGIDaemonProcess myproject.tld user=you group=you WSGIReloadMechanism Process WSGIScriptAlias / /var/www/myproject/apache/myapp.py you just have to touch myapp.py to force reload the subinterpreter(s). Unfortunately, I'm running Debian Etch, which has an older mod_wsgi, which does not have the process reloading mechanism. But I guess if this gets too painful I'll upgrade. rg -- http://mail.python.org/mailman/listinfo/python-list
More mod_wsgi weirdness: process restarts on redirect
I'm running mod_wsgi under apache (on Debian etch so it's a somewhat out of date version, though I doubt that has anything to do with this issue). I have a little test page that displays the process ID under which my app is running, and some global state, so I can tell when the wsgi app gets reloaded. This mostly only happens when I restart apache, but it also seems to happen when my WSGI app does an HTTP 302 redirect. (I'm actually using Yaro and calling req.redirect, but that only does a straightforward HTTP 302 redirect as far as I can tell.) It not only resets the global state, but changes process ID, so it seems to be doing a complete restart of mod_wsgi, which seems a little excessive. My question is: is this supposed to be happening? Or is this an indication that something is wrong, and if so, what? Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Re: More mod_wsgi weirdness: process restarts on redirect
In article rnospamon-a73662.15010729012...@news.gha.chartermi.net,
Ron Garret rnospa...@flownet.com wrote:
I'm running mod_wsgi under apache (on Debian etch so it's a somewhat out
of date version, though I doubt that has anything to do with this issue).
I have a little test page that displays the process ID under which my
app is running, and some global state, so I can tell when the wsgi app
gets reloaded. This mostly only happens when I restart apache, but it
also seems to happen when my WSGI app does an HTTP 302 redirect. (I'm
actually using Yaro and calling req.redirect, but that only does a
straightforward HTTP 302 redirect as far as I can tell.) It not only
resets the global state, but changes process ID, so it seems to be doing
a complete restart of mod_wsgi, which seems a little excessive.
My question is: is this supposed to be happening? Or is this an
indication that something is wrong, and if so, what?
Thanks,
rg
Here's a standalone WSGI app demonstrating the phenomenon:
def redirect_test(env, start):
if env['PATH_INFO']:
start('302 Found', [('Location', '/')])
return ['Redirecting']
else:
start('200 OK', [('Content-type', 'text/plain')])
return ['PID', str(os.getpid())]
pass
application = redirect_test
Fire this up under mod_wsgi and observe that the process ID stays the
same when you reload the app. Now add a path component to trigger the
redirect and observe that process ID changes.
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: More mod_wsgi weirdness: process restarts on redirect
In article mailman.8321.1233272610.3487.python-l...@python.org, Joshua Kugler jos...@joshuakugler.com wrote: Ron Garret wrote: My question is: is this supposed to be happening? Or is this an indication that something is wrong, and if so, what? You are probably just hitting a different instance of Apache, thus the different process ID. Yep, that's what it turned out to be. I thought I had a WSGIDaemonProcess processes=1 directive in my config, but I had it in the wrong place (a different vhost) so it wasn't actually active. But that leaves me wondering why the redirect would reliably trigger switching processes. The reason I thought that I had the correct configuration and only had one process is that when I reloaded the non-redirected page I *always* got the same process ID. How does mod_wsgi decide which process (and which thread for that matter) to use? rg -- http://mail.python.org/mailman/listinfo/python-list
Sloooooowwwww WSGI restart
I'm running a WSGI app under apache/mod_wsgi and I've noticed that whenever I restart the server after making a code change it takes a very long time (like a minute) before the script is active again. In other words, I do an apachectl restart, reload the page in my browser, and one minute later it finally comes up. During this time CPU usage is essentially zero. Loading all the code manually into a python interpreter is virtually instantaneous, and all subsequence interactions with the app are very fast. Does anyone have any ideas what might be going on or how to debug this? Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Sloooooowwwww WSGI restart
In article mailman.8228.1233179089.3487.python-l...@python.org, Aleksandar Radulovic a...@a13x.net wrote: Hi there, On Wed, Jan 28, 2009 at 9:35 PM, Ron Garret rnospa...@flownet.com wrote: I'm running a WSGI app under apache/mod_wsgi and I've noticed that Off the bat, there's no reason to run an app under apache/mod_wsgi while developing it, ie. if u use Pylons or TurboGears, there's an easier way to serve the app (using paster or cherrypy). Yes, I know. I guess I'm just a masochist. But this would be an issue in production as well. I don't want to have to go down for a minute every time I push a new release. Secondly, why are you restarting apache after code changes? In normal circumstances, you shouldn't have to do that. I thought (and experiment confirms) that only the main WSGI app file gets reloaded automatically when it changes, not the libraries. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Sloooooowwwww WSGI restart
In article mailman.8227.1233179051.3487.python-l...@python.org, Jean-Paul Calderone exar...@divmod.com wrote: On Wed, 28 Jan 2009 13:35:56 -0800, Ron Garret rnospa...@flownet.com wrote: I'm running a WSGI app under apache/mod_wsgi and I've noticed that whenever I restart the server after making a code change it takes a very long time (like a minute) before the script is active again. In other words, I do an apachectl restart, reload the page in my browser, and one minute later it finally comes up. During this time CPU usage is essentially zero. Loading all the code manually into a python interpreter is virtually instantaneous, and all subsequence interactions with the app are very fast. Does anyone have any ideas what might be going on or how to debug this? strace is nice. It sounds like some blocking network operation. rdns can often cause things like this. If you're lucky, strace will point right at the problem. Jean-Paul That's a good idea. Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Sloooooowwwww WSGI restart
In article 4cd232ff-ba8b-47aa-8ee6-d8d9712db...@s1g2000prg.googlegroups.com, Graham Dumpleton graham.dumple...@gmail.com wrote: On Jan 29, 8:35 am, Ron Garret rnospa...@flownet.com wrote: I'm running a WSGI app under apache/mod_wsgiand I've noticed that whenever I restart the server after making a code change it takes a very long time (like a minute) before the script is active again. In other words, I do an apachectl restart, reload the page in my browser, and one minute later it finally comes up. During this time CPU usage is essentially zero. Loading all the code manually into a python interpreter is virtually instantaneous, and all subsequence interactions with the app are very fast. Does anyone have any ideas what might be going on or how to debug this? The better place to discuss this is the mod_wsgi list. http://groups.google.com/group/modwsgi?hl=en The reason I don't go there is that I'm using an NNTP client. I find Google Groups' web interface to be very annoying because there's no way to keep track of read messages. As to the problem, you need to distinguish between whether it is Apache that is taking a long time to restart and ready to handle requests, or whether the delay is on the first subsequent request made against your WSGI application. When Apache restarts, it doesn't by default load your WSGI application, it only does that the first time a request comes in directed at it. Thus, if after you restart Apache you do a request of a static file, do you get a response straight away? I thought I did, but having poked at it more I now realize that I was just getting a cached page. It does appear to be an apache issue, not a WSGI issue. Sorry for the false alarm. rg -- http://mail.python.org/mailman/listinfo/python-list
Yaro vs WebOb
I'm selecting infrastructure for a web development and I've found two lightweight frameworks that seem to offer a lot of bang-for-the-byte: Yaro and WebOb. I'm wondering if anyone here has used either or both and has opinions about them. What has been your experience with them? Which do you prefer? Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
wsgi silently swallows errors
Consider the following wsgi app:
def application(env, start_response):
start_response('200 OK',[('Content-type','text/plain')])
yield hello
x=1/0
yield world
The result of this is that the web browser displays hello and an error
message ends up in the web log. But there is no other indication that
an error has occurred.
Is there any way to get WSGI to not silently swallow errors that occur
after start_response has been called?
Thanks,
rg
--
http://mail.python.org/mailman/listinfo/python-list
WSGI question: reading headers before message body has been read
I'm writing a WSGI application and I would like to check the content-
length header before reading the content to make sure that the content
is not too big in order to prevent denial-of-service attacks. So I do
something like this:
def application(environ, start_response):
status = 200 OK
headers = [('Content-Type', 'text/html'), ]
start_response(status, headers)
if int(environ['CONTENT_LENGTH'])1000: return 'File too big'
But this doesn't seem to work. If I upload a huge file it still waits
until the entire file has been uploaded before complaining that it's
too big.
Is it possible to read the HTTP headers in WSGI before the request
body has been read?
Thanks,
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: WSGI question: reading headers before message body has been read
On Jan 18, 11:29 am, Diez B. Roggisch de...@nospam.web.de wrote:
Ron Garret schrieb:
I'm writing a WSGI application and I would like to check the content-
length header before reading the content to make sure that the content
is not too big in order to prevent denial-of-service attacks. So I do
something like this:
def application(environ, start_response):
status = 200 OK
headers = [('Content-Type', 'text/html'), ]
start_response(status, headers)
if int(environ['CONTENT_LENGTH'])1000: return 'File too big'
But this doesn't seem to work. If I upload a huge file it still waits
until the entire file has been uploaded before complaining that it's
too big.
Is it possible to read the HTTP headers in WSGI before the request
body has been read?
AFAIK that is nothing that WSGI defines - it's an implementation-detail
of your server. Which one do you use?
Apache at the moment, with lighttpd as a contender to replace it.
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: WSGI question: reading headers before message body has been read
On Jan 18, 11:43 am, Petite Abeille petite.abei...@gmail.com wrote:
On Jan 18, 2009, at 8:01 PM, Ron Garret wrote:
def application(environ, start_response):
status = 200 OK
headers = [('Content-Type', 'text/html'), ]
start_response(status, headers)
if int(environ['CONTENT_LENGTH'])1000: return 'File too big'
How would that work for chunked transfer-encoding?
It wouldn't. But many clients don't use chunked-transfer-encoding
when uploading files whose size is known. In that case it would be
nice to let users know that their upload is going to fail BEFORE they
waste hours waiting for 10GB of data to go down the wire.
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: WSGI question: reading headers before message body has been read
On Jan 18, 12:40 pm, Diez B. Roggisch de...@nospam.web.de wrote:
Ron Garret schrieb:
On Jan 18, 11:29 am, Diez B. Roggisch de...@nospam.web.de wrote:
Ron Garret schrieb:
I'm writing a WSGI application and I would like to check the content-
length header before reading the content to make sure that the content
is not too big in order to prevent denial-of-service attacks. So I do
something like this:
def application(environ, start_response):
status = 200 OK
headers = [('Content-Type', 'text/html'), ]
start_response(status, headers)
if int(environ['CONTENT_LENGTH'])1000: return 'File too big'
But this doesn't seem to work. If I upload a huge file it still waits
until the entire file has been uploaded before complaining that it's
too big.
Is it possible to read the HTTP headers in WSGI before the request
body has been read?
AFAIK that is nothing that WSGI defines - it's an implementation-detail
of your server. Which one do you use?
Apache at the moment, with lighttpd as a contender to replace it.
Together with mod_wsgi?
Diez
Yes. (Is there any other way to run WSGI apps under Apache?)
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: WSGI question: reading headers before message body has been read
On Jan 18, 1:21 pm, Graham Dumpleton graham.dumple...@gmail.com
wrote:
On Jan 19, 6:01 am, Ron Garret r...@flownet.com wrote:
I'm writing a WSGI application and I would like to check the content-
length header before reading the content to make sure that the content
is not too big in order to prevent denial-of-service attacks. So I do
something like this:
def application(environ, start_response):
status = 200 OK
headers = [('Content-Type', 'text/html'), ]
start_response(status, headers)
if int(environ['CONTENT_LENGTH'])1000: return 'File too big'
You should be returning 413 (Request Entity Too Large) error status
for that specific case, not a 200 response.
You should not be returning a string as response content as it is very
inefficient, wrap it in an array.
But this doesn't seem to work. If I upload a huge file it still waits
until the entire file has been uploaded before complaining that it's
too big.
Is it possible to read the HTTP headers in WSGI before the request
body has been read?
Yes.
The issue is that in order to avoid the client sending the data the
client needs to actually make use of HTTP/1.1 headers to indicate it
is expecting a 100-continue response before sending data. You don't
need to handle that as Apache/mod_wsgi does it for you, but the only
web browser I know of that supports 100-continue is Opera browser.
Clients like curl do also support it as well though. In other words,
if people use IE, Firefox or Safari, the request content will be sent
regardless anyway.
There is though still more to this though. First off is that if you
are going to handle 413 errors in your own WSGI application and you
are using mod_wsgi daemon mode, then request content is still sent by
browser regardless, even if using Opera. This is because the act of
transferring content across to mod_wsgi daemon process triggers return
of 100-continue to client and so it sends data. There is a ticket for
mod_wsgi to implement proper 100-continue support for daemon mode, but
will be a while before that happens.
Rather than have WSGI application handle 413 error cases, you are
better off letting Apache/mod_wsgi handle it for you. To do that all
you need to do is use the Apache 'LimitRequestBody' directive. This
will check the content length for you and send 413 response without
the WSGI application even being called. When using daemon mode, this
is done in Apache child worker processes and for 100-continue case
data will not be read at all and can avoid client sending it if using
Opera.
Only caveat on that is the currently available mod_wsgi has a bug in
it such that 100-continue requests not always working for daemon mode.
You need to apply fix in:
http://code.google.com/p/modwsgi/issues/detail?id=121
For details on LimitRequestBody directive see:
http://httpd.apache.org/docs/2.2/mod/core.html#limitrequestbody
Graham
Thanks for the detailed response!
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: Need help getting MoinMoin to run under WSGI
In article rnospamon-79fa22.21571527122...@news.gha.chartermi.net, Ron Garret rnospa...@flownet.com wrote: I successfully installed MoinMoin as a CGI according to the instructions on the moinmo.in site. But when I tried to switch over to running it under wsgi it failed thusly: [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] Traceback (most recent call last): [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] File /www/wikis/genesisgroup/moin.wsgi, line 49, in ? [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] from MoinMoin.server.server_wsgi import WsgiConfig, moinmoinApp [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] ImportError: No module named MoinMoin.server.server_wsgi The problem, I believe, is that I have both Python 2.4 and 2.5 installed (it's a Debian box) and MM is installed under 2.5 but WSGI is using 2.4. I tried to fix this by setting WSGIPythonHome but to no avail. I can't figure out what to set it to. The instructions say: the WSGIPythonHome directive should be used to specify the exact location of the Python installation corresponding to the version of Python compiled against I have two problems with this. First, I didn't compile mod_wsgi, I got it pre-built as a Debian module. Second, what does the exact location of the Python installation even mean? Python2.5 is spread out in at least three different places: /usr/local/bin, /usr/lib/python2.5, and /usr/local/lib/python2.5. I've tried setting WSGIPythonHome to all of those (and a few other things as well) and nothing worked. Also, the version of Python compiled against seems very odd. What does that mean? Surely I don't have to recompile mod_wsgi every time I change to a new version of Python? Help! Thanks! rg So never mind, I figured it out. I did indeed have to recompile mod_wsgi from source to get it to use Python 2.5. (That turned out to be a major hassle. I had to do it twice. The first time through it made Apache dump core. I still don't know why.) Seems to be working now though. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Need help getting MoinMoin to run under WSGI
In article f97e6514-1d36-42ed-b0f7-98e2b3162...@d36g2000prf.googlegroups.com, Graham Dumpleton graham.dumple...@gmail.com wrote: On Dec 28, 7:22 pm, Ron Garret rnospa...@flownet.com wrote: In article rnospamon-79fa22.21571527122...@news.gha.chartermi.net, Ron Garret rnospa...@flownet.com wrote: I successfully installed MoinMoin as a CGI according to the instructions on the moinmo.in site. But when I tried to switch over to running it under wsgi it failed thusly: [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] Traceback (most recent call last): [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] File /www/wikis/genesisgroup/moin.wsgi, line 49, in ? [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] from MoinMoin.server.server_wsgi import WsgiConfig, moinmoinApp [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] ImportError: No module named MoinMoin.server.server_wsgi The problem, I believe, is that I have both Python 2.4 and 2.5 installed (it's a Debian box) and MM is installed under 2.5 but WSGI is using 2.4. I tried to fix this by setting WSGIPythonHome but to no avail. I can't figure out what to set it to. The instructions say: the WSGIPythonHome directive should be used to specify the exact location of the Python installation corresponding to the version of Python compiled against I have two problems with this. First, I didn't compilemod_wsgi, I got it pre-built as a Debian module. Second, what does the exact location of the Python installation even mean? Python2.5 is spread out in at least three different places: /usr/local/bin, /usr/lib/python2.5, and /usr/local/lib/python2.5. I've tried setting WSGIPythonHome to all of those (and a few other things as well) and nothing worked. Also, the version of Python compiled against seems very odd. What does that mean? Surely I don't have to recompilemod_wsgievery time I change to a new version of Python? Help! Thanks! rg So never mind, I figured it out. I did indeed have to recompilemod_wsgifrom source to get it to use Python 2.5. (That turned out to be a major hassle. I had to do it twice. The first time through it made Apache dump core. I still don't know why.) Seems to be working now though. It probably crashed the first time because you didn't do a full stop of Apache and instead just did a reload. Doing a reload may not unload the Python libraries from Apache process correctly and so would have been a mix of code for different versions of Python in same process. Graham Yes, I don't specifically recall exactly what happened now, but that seems likely. Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Need help getting MoinMoin to run under SCGI
So I have a MoinMoin installation running as a cgi and also under wsgi. Since I was on a roll I decided to press my luck and try running it under scgi. Following a suggestion in the following article: http://www.linuxjournal.com/article/9310 I wrote this little server adapter: from MoinMoin.server.server_cgi import CgiConfig, run class Config(CgiConfig): name = 'moin' import scgi import scgi.scgi_server class MoinHandler(scgi.scgi_server.SCGIHandler): def produce_cgilike(self, env, bodysize): run(Config) server = scgi.scgi_server.SCGIServer( handler_class=MoinHandler, port=4000 ) server.serve() It works -- sort of. I can fire it up and get to the login page, but it won't actually let me log in. I'm guessing this is because I haven't munged the CGI form data properly. I thought I'd ask if anyone with experience using SCGI can just tell me what I'm doing wrong before I dive into debugging this. Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Need help getting MoinMoin to run under WSGI
I successfully installed MoinMoin as a CGI according to the instructions on the moinmo.in site. But when I tried to switch over to running it under wsgi it failed thusly: [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] Traceback (most recent call last): [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] File /www/wikis/genesisgroup/moin.wsgi, line 49, in ? [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] from MoinMoin.server.server_wsgi import WsgiConfig, moinmoinApp [Sat Dec 27 21:44:14 2008] [error] [client 66.214.189.2] ImportError: No module named MoinMoin.server.server_wsgi The problem, I believe, is that I have both Python 2.4 and 2.5 installed (it's a Debian box) and MM is installed under 2.5 but WSGI is using 2.4. I tried to fix this by setting WSGIPythonHome but to no avail. I can't figure out what to set it to. The instructions say: the WSGIPythonHome directive should be used to specify the exact location of the Python installation corresponding to the version of Python compiled against I have two problems with this. First, I didn't compile mod_wsgi, I got it pre-built as a Debian module. Second, what does the exact location of the Python installation even mean? Python2.5 is spread out in at least three different places: /usr/local/bin, /usr/lib/python2.5, and /usr/local/lib/python2.5. I've tried setting WSGIPythonHome to all of those (and a few other things as well) and nothing worked. Also, the version of Python compiled against seems very odd. What does that mean? Surely I don't have to recompile mod_wsgi every time I change to a new version of Python? Help! Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: CTypes on a 64 bit machine truncates pointers to 32 bits?
In article [EMAIL PROTECTED],
Martin v. Löwis [EMAIL PROTECTED] wrote:
foolib = cdll.LoadLibrary('foo.so')
foolib.foo(0xF)
0
Shouldn't you tell ctypes that the argument and the result
of foo is a pointer?
Yeah... that's it. Default return type is int, which I assumed would be
64 bits on a 64 bit machine, but turns out it's 32. Stupid.
rg
--
http://mail.python.org/mailman/listinfo/python-list
CTypes on a 64 bit machine truncates pointers to 32 bits?
CTypes on a 64-bit machine appears to be truncating pointers to 32 bits:
[EMAIL PROTECTED]:~]$ uname -a
Linux monster1 2.6.18-6-amd64 #1 SMP Mon Jun 16 22:30:01 UTC 2008 x86_64
GNU/Linux
[EMAIL PROTECTED]:~]$ cat foo.c
void* foo(void* x) { return x; }
[EMAIL PROTECTED]:~]$ gcc -fPIC -shared foo.c -o foo.so
[EMAIL PROTECTED]:~]$ file foo.so
foo.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not
stripped
[EMAIL PROTECTED]:~]$ python
Python 2.5 (release25-maint, Jul 23 2008, 18:15:29)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type help, copyright, credits or license for more information.
from ctypes import *
foolib = cdll.LoadLibrary('foo.so')
foolib.foo(0xF)
0
sizeof(c_void_p)
8
I am aware of http://bugs.python.org/issue1703286 but I get the same
result in Python2.5.2:
[EMAIL PROTECTED]:~]$ py252/bin/python
Python 2.5.2 (r252:60911, Sep 18 2008, 10:48:29)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type help, copyright, credits or license for more information.
from ctypes import *
foolib = cdll.LoadLibrary('foo.so')
foolib.foo(0xF)
0
sizeof(c_void_p)
8
--
http://mail.python.org/mailman/listinfo/python-list
Re: Parsing MIME-encoded data in an HTTP request
In article [EMAIL PROTECTED],
Michael Ströder [EMAIL PROTECTED] wrote:
Ron Garret wrote:
In article [EMAIL PROTECTED],
Ron Garret [EMAIL PROTECTED] wrote:
In article [EMAIL PROTECTED],
Michael Ströder [EMAIL PROTECTED] wrote:
Ron Garret wrote:
I'm writing a little HTTP server and need to parse request content that
is mime-encoded. All the MIME routines in the Python standard library
seem to have been subsumed into the email package, which makes this
operation a little awkward.
How about using cgi.parse_multipart()?
Unfortunately cgi.parse_multipart doesn't handle nested multiparts,
which the requests I'm getting have. You have to use a FieldStorage
object to do that, and that only works if you're actually in a cgi
environment, which I am not. The server responds to these requests
directly.
Anyway, thanks for the idea.
Hm, it actually seems to work if I manually pass in the outerboundary
parameter and environ={'REQUEST_METHOD':'POST'} That seems like the
Right Answer.
I'm also using it to parse form parameters in a message body received by
POST.
CIao, Michael.
Just for the record, here's the incantation I ended up with:
class post_handler(BaseHTTPRequestHandler):
def do_POST(self):
form = cgi.FieldStorage(fp=self.rfile, headers=self.headers,
environ={'REQUEST_METHOD':'POST'})
...
works like a charm.
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: Parsing MIME-encoded data in an HTTP request
In article [EMAIL PROTECTED], [EMAIL PROTECTED] wrote: On Jul 3, 3:59 pm, Ron Garret [EMAIL PROTECTED] wrote: I'm writing a little HTTP server and need to parse request content that is mime-encoded. All the MIME routines in the Python standard library seem to have been subsumed into the email package, which makes this operation a little awkward. To deal with messages of that kind, I've seen modules such as 'rfc822', and 'mimetools' (which apparently builds itself from 'rfc822', so it might be more complete). There's also 'mimetypes', in case you need to deal with file extensions and their corresponding MIME media type. From the mimetools docs: Deprecated since release 2.3. The email package should be used in preference to the module. This module is present only to maintain backward compatibility. It seems I have to do the following: 1. Extract the content-length header from the HTTP request and use that to read the payload. 2. Stick some artificial-looking headers onto the beginning of this payload to make it look like an email message (including the content-type and content-transfer-encoding headers) 3. Parse the resulting string into a email message Email? Why does an HTTP server need to build an email message? It shouldn't. That's my whole point. But see the docs excerpt above. I remember doing things like that some time ago when building an HTTP server myself (http://code.google.com/p/sws-d/). Incidentally, I resisted the urge to use much of the Python's library facilities (most things are done manually; am I a knucklehead or what!? :). You might wanna take a look to get some ideas. I'd much prefer not to reinvent this particular wheel. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Parsing MIME-encoded data in an HTTP request
In article [EMAIL PROTECTED], Michael Ströder [EMAIL PROTECTED] wrote: Ron Garret wrote: I'm writing a little HTTP server and need to parse request content that is mime-encoded. All the MIME routines in the Python standard library seem to have been subsumed into the email package, which makes this operation a little awkward. How about using cgi.parse_multipart()? Ciao, Michael. Unfortunately cgi.parse_multipart doesn't handle nested multiparts, which the requests I'm getting have. You have to use a FieldStorage object to do that, and that only works if you're actually in a cgi environment, which I am not. The server responds to these requests directly. Anyway, thanks for the idea. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Parsing MIME-encoded data in an HTTP request
In article [EMAIL PROTECTED],
Ron Garret [EMAIL PROTECTED] wrote:
In article [EMAIL PROTECTED],
Michael Ströder [EMAIL PROTECTED] wrote:
Ron Garret wrote:
I'm writing a little HTTP server and need to parse request content that
is mime-encoded. All the MIME routines in the Python standard library
seem to have been subsumed into the email package, which makes this
operation a little awkward.
How about using cgi.parse_multipart()?
Ciao, Michael.
Unfortunately cgi.parse_multipart doesn't handle nested multiparts,
which the requests I'm getting have. You have to use a FieldStorage
object to do that, and that only works if you're actually in a cgi
environment, which I am not. The server responds to these requests
directly.
Anyway, thanks for the idea.
rg
Hm, it actually seems to work if I manually pass in the outerboundary
parameter and environ={'REQUEST_METHOD':'POST'} That seems like the
Right Answer.
Woohoo!
Thanks Michael!
rg
--
http://mail.python.org/mailman/listinfo/python-list
Parsing MIME-encoded data in an HTTP request
I'm writing a little HTTP server and need to parse request content that is mime-encoded. All the MIME routines in the Python standard library seem to have been subsumed into the email package, which makes this operation a little awkward. It seems I have to do the following: 1. Extract the content-length header from the HTTP request and use that to read the payload. 2. Stick some artificial-looking headers onto the beginning of this payload to make it look like an email message (including the content-type and content-transfer-encoding headers) 3. Parse the resulting string into a email message That works, but it feels way too hackish for my tastes. Surely there must be a better/more standard way of doing this? Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to change the default string encoding?
In article [EMAIL PROTECTED], Peter Otten [EMAIL PROTECTED] wrote: If all else fails there's sys.setdefaultencoding(latin1) Andre\xe9 Ramel.decode() u'Andre\xe9 Ramel' but that's an evil hack, you should rather talk to the maintainer of the offending code to update it to accept unicode. Yes, but I need to hack around it until I can get it fixed. Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Is there a way to change the default string encoding?
Is there a way to change the default string encoding used by the string.encode() method? My default environment is utf-8 but I need it to be latin-1 to avoid errors like this: 'Andr\xe9 Ramel'.decode() Traceback (most recent call last): File stdin, line 1, in module UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 4: ordinal not in range(128) I can't change the code to pass an encoding argument to the decode method because it's someone else's code. Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Is wsgi ready for prime time?
The wsgiref module in Python 2.5 seems to be empty: [EMAIL PROTECTED]:~/Sites/modpy]$ python Python 2.5 (r25:51908, Mar 1 2007, 10:09:05) [GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin Type help, copyright, credits or license for more information. import wsgiref dir(wsgiref) ['__builtins__', '__doc__', '__file__', '__name__', '__path__'] So... is wsgi considered ready for production use, or is it still on the bleeding edge? And if the former, which implementation should one use? rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Is wsgi ready for prime time?
In article [EMAIL PROTECTED], Stargaming [EMAIL PROTECTED] wrote: Ron Garret wrote: The wsgiref module in Python 2.5 seems to be empty: [EMAIL PROTECTED]:~/Sites/modpy]$ python Python 2.5 (r25:51908, Mar 1 2007, 10:09:05) [GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin Type help, copyright, credits or license for more information. import wsgiref dir(wsgiref) ['__builtins__', '__doc__', '__file__', '__name__', '__path__'] So... is wsgi considered ready for production use, or is it still on the bleeding edge? And if the former, which implementation should one use? rg help(wsgiref) Help on package wsgiref: NAME wsgiref - wsgiref -- a WSGI (PEP 333) Reference Library DESCRIPTION Current Contents: * util -- Miscellaneous useful functions and wrappers * headers -- Manage response headers * handlers -- base classes for server/gateway implementations * simple_server -- a simple BaseHTTPServer that supports WSGI * validate -- validation wrapper that sits between an app and a server to detect errors in either To-Do: * cgi_gateway -- Run WSGI apps under CGI (pending a deployment standard) * cgi_wrapper -- Run CGI apps under WSGI * router -- a simple middleware component that handles URL traversal PACKAGE CONTENTS handlers headers simple_server util validate Reading the documentation can be useful sometimes. Recommending http://docs.python.org/lib/module-wsgiref.html, too. I did read the documentation, but the documentation does not seem to reflect reality, e.g.: wsgiref.util Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'module' object has no attribute 'util' wsgiref.headers Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'module' object has no attribute 'headers' wsgiref.handlers Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'module' object has no attribute 'handlers' Hence my question. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Select weirdness
In article [EMAIL PROTECTED], Irmen de Jong [EMAIL PROTECTED] wrote: Ron Garret wrote: I don't understand why socketserver calling select should matter. (And BTW, there are no calls to select in SocketServer.py. I'm using Python2.5.) You don't *need* a select at all. Yes I do, because what I'm really writing is a dispatching proxy that has to serve many simultaneous connections. Here's the full story in case you're interested: We have an application that is currently fielded as a cgi. We have a development server that needs to run multiple copies of the application at the same time. This is so that developers can push changes into their private sandboxes for evaluation before going into the main development branch. Each sandbox has its own source tree, its own database, and its own URL namespace. There are a small number of URLs in the application that are performance bottlenecks (they are used to serve AJAX updates). In order to alleviate that bottleneck without having to rewrite the whole application to run under mod_python or some such thing we've written a special dedicated server that handles only the AJAX requests. The tricky part is that each developer needs to have their own copy of this server running because each developer can have different code that needs to run to serve those requests. Assigning each developer a dedicated IP port would be a configuration nightmare, so these servers serve run on unix sockets rather than TCP sockets. I have not been able to find a proxy server that can proxy to unix sockets, so I need to write my own. Conceptually its a very simple thing: read the first line of an HTTP request, parse it with a regexp to extract the sandbox name, connect to the appropriate unix server socket, and then bidirectionally pipe bytes back and forth. But it has to do this for multiple connections simultaneously, which is why I need select. Anyway, try the following instead: That won't work for POST requests. Why not? Because POST requests can be very complicated. Just add some more code to deal with the POST request body. I was really hoping to avoid having to write a fully HTTP-aware proxy. There should be a content-length header to tell you how many bytes to read after the header section has finished. Not if the content-transfer-encoding is chunked. Or if there are multiple file attachments. Also, even GET requests can become very complicated (from a protocol point of view) in HTTP 1.1. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Select weirdness
In article [EMAIL PROTECTED], Gabriel Genellina [EMAIL PROTECTED] wrote: En Mon, 23 Apr 2007 04:33:22 -0300, Ron Garret [EMAIL PROTECTED] escribió: I have not been able to find a proxy server that can proxy to unix sockets, so I need to write my own. Conceptually its a very simple thing: read the first line of an HTTP request, parse it with a regexp to extract the sandbox name, connect to the appropriate unix server socket, and then bidirectionally pipe bytes back and forth. But it has to do this for multiple connections simultaneously, which is why I need select. No. This is what the *server* should do, not the *handler*. The server listens on incoming requests, and creates a new handler for each one. Inside a handler, you don't have to worry about multiple connections. If you want to process simultaneous connections, inherit your server from ThreadingMixIn or ForkingMixIn (or use one of the predefined server classes). Ah, good point. But that still leaves the problem of keep-alive connections. I still need to be able to forward data in both directions without knowing ahead of time when it will arrive. (I think I can solve the problem by using send and recv directly BTW.) rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Select weirdness
In article [EMAIL PROTECTED], Jean-Paul Calderone [EMAIL PROTECTED] wrote: Twisted does this out of the box, for what it's worth. Thanks. I will look at that. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Select weirdness
In article [EMAIL PROTECTED], Donn Cave [EMAIL PROTECTED] wrote: In article [EMAIL PROTECTED], Ron Garret [EMAIL PROTECTED] wrote: The answer is obvious: select is looking only at the underlying socket, and not at the rfile buffers. So... is this a bug in select? Or a bug in my code? Yes. I don't see any specific followup to this point, but it is or at least should be a well known limitation of C library I/O and select. select() is an operating system function that has no way to know the state of your process I/O buffers, nor do the C stdio buffers don't come with a standard way to inspect that state. Therefore, if you mix C I/O with select(), you're more or less out of luck. This applies directly to Python, because it calls the operating system select() function and it uses C stdio buffers for its file object (and entices you to make this mistake by supporting a file object as an input to select().) This conflict can be relieved after a fashion by eliminating the buffer. The now unbuffered C fgets / Python readline won't store extra lines that select can't see, but the semantics of the operation are still a poor fit with the usual application of select. The number of system-level I/O calls is significantly greater as you transfer data one byte at a time, which may add a lot of overhead if you transfer a lot of data, and your readline() function still can't return until it gets that '\n', so a half line can block your application. It isn't a lot of work to read data with operating system functions that are compatible with select - os.read(), socket.recv() - and break it up into lines on your own, and this completely and efficiently resolves the problem. Yep, this is pretty much the conclusion I've come to. As an editorial comment, I think it's quite confusing for Python's select to accept file objects without complaint if it's not going to do the Right Thing with them. rg -- http://mail.python.org/mailman/listinfo/python-list
Select weirdness
Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):
from SocketServer import *
from socket import *
from select import select
class myHandler(StreamRequestHandler):
def handle(self):
print ''
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.readline()
if len(l)3: break
print l,
pass
pass
printself.wfile, 'HTTP/1.0 200 OK'
printself.wfile, 'Content-type: text/plain'
printself.wfile
printself.wfile, 'foo'
self.rfile.close()
self.wfile.close()
print ''
pass
pass
def main():
server = TCPServer(('',8080), myHandler)
server.serve_forever()
pass
if __name__ == '__main__': main()
If I telnet into this server and type in an HTTP request manually it
works fine. But when I try to access this with a browser (I've tried
Firefox and Safari -- both do the same thing) it hangs immediately after
reading the first line in the request (i.e. before reading the first
header).
When I click the stop button in the browser it breaks the logjam and
the server reads the headers (but then of course it dies trying to write
the response to a now-closed socket).
The only difference I can discern is that the browser send \r\n for
end-of-line while telnet just sends \n. But I don't see why that should
make any difference. So I'm stumped. Any clues would be much
appreciated.
Thanks,
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: Select weirdness
In article [EMAIL PROTECTED], Bjoern Schliessmann [EMAIL PROTECTED] wrote: The only difference I can discern is that the browser send \r\n for end-of-line while telnet just sends \n. ... But I don't see why that should make any difference. Easy. If you only accept \r\n as EOL, you'll wait forever unless you really receive it. But it's SELECT that is hanging. Select knows (or ought to know) nothing of end-of-line markers. Readline (when it is called) is doing the Right Thing, and my end-of-request test is (currently) reading a line less than 3 characters long. All that is working. The problem is that select is saying there is no input (and therefore hanging) when in fact there is input. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Select weirdness
In article [EMAIL PROTECTED], Irmen de Jong [EMAIL PROTECTED] wrote: Ron Garret wrote: Here's my code. It's a teeny weeny little HTTP server. (I'm not really trying to reinvent the wheel here. What I'm really doing is writing a dispatching proxy server, but this is the shortest way to illustrate the problem I'm having): from SocketServer import * from socket import * from select import select class myHandler(StreamRequestHandler): def handle(self): print '' while 1: sl = select([self.rfile],[],[])[0] if sl: l = self.rfile.readline() if len(l)3: break print l, pass pass printself.wfile, 'HTTP/1.0 200 OK' printself.wfile, 'Content-type: text/plain' printself.wfile printself.wfile, 'foo' self.rfile.close() self.wfile.close() print '' pass pass You shouldn't use a select() of your own inside the handle method. The socketserver takes care of that for you. I don't understand why socketserver calling select should matter. (And BTW, there are no calls to select in SocketServer.py. I'm using Python2.5.) What you need to do is read the input from the socket until you've reached the end-of-input marker. That marker is an empty line containing just \r\n when dealing with HTTP GET requests. Any reason don't want to use the BasicHTTPServer that comes with Python? Yes, but it's a long story. What I'm really doing is writing a dispatching proxy. It reads the HTTP request and then redirects it to a number of different servers depending on the URL. The servers are all local and served off of unix sockets, not TCP sockets (which is why I can't just configure Apache to do this). The reason I'm running this weird configuration (in case you're wondering) is because this is a development machine and multiple copies of the same website run on it simultaneously. Each copy of the site runs a customized server to handle AJAX requests efficiently. Anyway, try the following instead: That won't work for POST requests. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Select weirdness
In article [EMAIL PROTECTED],
Ron Garret [EMAIL PROTECTED] wrote:
Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):
from SocketServer import *
from socket import *
from select import select
class myHandler(StreamRequestHandler):
def handle(self):
print ''
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.readline()
if len(l)3: break
print l,
pass
pass
printself.wfile, 'HTTP/1.0 200 OK'
printself.wfile, 'Content-type: text/plain'
printself.wfile
printself.wfile, 'foo'
self.rfile.close()
self.wfile.close()
print ''
pass
pass
def main():
server = TCPServer(('',8080), myHandler)
server.serve_forever()
pass
if __name__ == '__main__': main()
If I telnet into this server and type in an HTTP request manually it
works fine. But when I try to access this with a browser (I've tried
Firefox and Safari -- both do the same thing) it hangs immediately after
reading the first line in the request (i.e. before reading the first
header).
When I click the stop button in the browser it breaks the logjam and
the server reads the headers (but then of course it dies trying to write
the response to a now-closed socket).
The only difference I can discern is that the browser send \r\n for
end-of-line while telnet just sends \n. But I don't see why that should
make any difference. So I'm stumped. Any clues would be much
appreciated.
I have reproduced the problem using Telnet, so that proves it's not an
EOL issue. The problem seems to be timing-related. If I type the
request in manually it works. If I paste it in all at once, it hangs.
It's actually even weirder than that: if I pipe the request into a
telnet client then it hangs after the request line, just with a browser
(or wget). But if I literally paste the request into a window running a
telnet client, it gets past the request line and four out of seven
headers before it hangs.
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: Select weirdness
I think I've figured out what's going on. First, here's the smoking gun: I changed the code as follows: class myHandler(StreamRequestHandler): def handle(self): print '' while 1: sl = select([self.rfile],[],[],1)[0] print sl l = self.rfile.readline() if len(l)3: break print l, pass (Rest of code is unchanged.) In other words, I select on the rfile object and added a timeout. Here's the result when I cut-and-paste an HTTP request: [socket._fileobject object at 0x6b110] GET /foo/baz HTTP/1.1 [socket._fileobject object at 0x6b110] Accept: */* [socket._fileobject object at 0x6b110] Accept-Language: en [socket._fileobject object at 0x6b110] Accept-Encoding: gzip, deflate [socket._fileobject object at 0x6b110] Cookie: c1=18:19:55.042196; c2=18:19:55.042508 [] User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/419 (KHTM [] L, like Gecko) Safari/419.3 [] Connection: keep-alive [] Host: localhost:8081 [] As you can see, the select call shows input available for a while (five lines) and then shows no input available despite the fact that there is manifestly still input available. The answer is obvious: select is looking only at the underlying socket, and not at the rfile buffers. So... is this a bug in select? Or a bug in my code? rg -- http://mail.python.org/mailman/listinfo/python-list
Bug in select (was: Re: Select weirdness)
In article [EMAIL PROTECTED],
Ron Garret [EMAIL PROTECTED] wrote:
The answer is obvious: select is looking only at the underlying socket,
and not at the rfile buffers.
Here is conclusive proof that there's a bug in select:
from socket import *
from select import select
s=socket(AF_INET, SOCK_STREAM)
s.bind(('',8080))
s.listen(5)
f = s.accept()[0].makefile()
# Now telnet to port 8080 and enter two lines of random text
f.readline()
select([f],[],[],1)
f.readline()
Here's the sample input:
[EMAIL PROTECTED]:~/devel/sockets]$ telnet localhost 8081
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
123
321
And this is the result:
f.readline()
'123\r\n'
select([f],[],[],1)
# After one second...
([], [], [])
f.readline()
'321\r\n'
So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bug in select
In article [EMAIL PROTECTED],
Erik Max Francis [EMAIL PROTECTED] wrote:
Ron Garret wrote:
So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?
It's hard to see how this demonstrates a bug in anything, since you're
telnetting to the wrong port in your example.
Geez you people are picky. Since I ran this several times I ran into
the TIM_WAIT problem. Here's the actual transcript:
Python 2.5 (r25:51908, Mar 1 2007, 10:09:05)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type help, copyright, credits or license for more information.
from socket import *
from select import select
s=socket(AF_INET, SOCK_STREAM)
s.bind(('',8080))
Traceback (most recent call last):
File stdin, line 1, in module
File string, line 1, in bind
socket.error: (48, 'Address already in use')
s.bind(('',8081))
s.listen(5)
f = s.accept()[0].makefile()
f.readline()
'123\r\n'
select([f],[],[],1)
([], [], [])
f.readline()
'321\r\n'
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bug in select
In article [EMAIL PROTECTED], Erik Max Francis [EMAIL PROTECTED] wrote: Ron Garret wrote: Geez you people are picky. Since I ran this several times I ran into the TIM_WAIT problem. Here's the actual transcript: It's not about being picky, it's about making it clear what your problem is. You're now describing an entirely different problem, Nope, it's been the same problem all along. If you don't think so then you have misunderstood something. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Bug in select (was: Re: Select weirdness)
In article [EMAIL PROTECTED], Dennis Lee Bieber [EMAIL PROTECTED] wrote: Well, on WinXP, Python 2.4, with I should have specified: I'm running 2.5 on unix. (I've reproduced the problem on both Linux and OS X.) rg -- http://mail.python.org/mailman/listinfo/python-list
Re: BaseHTTPServer and Apache
In article [EMAIL PROTECTED], Luis M. González [EMAIL PROTECTED] wrote: On Apr 13, 8:44 pm, Ron Garret [EMAIL PROTECTED] wrote: In article [EMAIL PROTECTED], Ron Garret [EMAIL PROTECTED] wrote: Does anyone know of a straightforward way to get Apache to forward requests to a given path to another HTTP server running on a different port? Never mind, I think I figured it out. Apparently what I need is the ProxyPassReverse directive. I'd still be interested in hearing about people's experience using BaseHTTPServer for real applications. Thanks, rg Some python web frameworks use the aproach you described by means of mod_rewrite. For example, Karrigell and Cherrypy (although they offer also other ways of deployment). This page in Karrigell's docs show how: http://karrigell.sourceforge.net/en/apache.htm Thanks for that pointer. Using the 502 handler to start the server is a nifty trick. So I've got all that working now. Next question: how do I set things up so that I can communicate to my server through a unix domain socket instead of a TCP socket? (The reason I want to do this is that I'm running multiple copies of this server so that multiple engineers can do development each in their own sandbox, and assigning each of them their own port number is a configuration nightmare.) I got as far as rewriting the server code to use a unix socket, but then I got stuck. rg -- http://mail.python.org/mailman/listinfo/python-list
BaseHTTPServer and Apache
I have a fairly large web app written in Python as a CGI fairly elaborate CGI. All of the requests go through a single CGI script which does authentication and session management and then dispatches to one of a number of handlers that generate the various pages. There is one page that is a performance bottleneck (because it is accessed automatically once a second by an XMLHTTPRequest, but that's another story). I have fixed this by re-implementing that page using a BaseHTTPServer. So I can fix my problem by redirecting the XMLHTTPRequest to a URL that is served by this BaseHTTPServer. The problem is that this server can't run on port 80 because Apache has to be there to serve everything else, and having a Python process serving a publicly accessible TCP port makes me a little queasy. Does anyone know of a straightforward way to get Apache to forward requests to a given path to another HTTP server running on a different port? Also, is anyone out there using BaseHTTPServer in an industrial setting? What has been your experience with it? And how do you make it fit in to your overall architecture? Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: BaseHTTPServer and Apache
In article [EMAIL PROTECTED], Ron Garret [EMAIL PROTECTED] wrote: Does anyone know of a straightforward way to get Apache to forward requests to a given path to another HTTP server running on a different port? Never mind, I think I figured it out. Apparently what I need is the ProxyPassReverse directive. I'd still be interested in hearing about people's experience using BaseHTTPServer for real applications. Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Python on a mac: how to build pythonw?
I'm trying to run the Python examples distributed with XCode and they all give me the same error: Traceback (most recent call last): File checktext.py, line 35, in module main() File checktext.py, line 8, in main pathname = EasyDialogs.AskFileForOpen(message='File to check end-of-lines in:') File /usr/local/lib/python2.5/plat-mac/EasyDialogs.py, line 650, in AskFileForOpen _interact() File /usr/local/lib/python2.5/plat-mac/EasyDialogs.py, line 53, in _interact AE.AEInteractWithUser(5000) MacOS.Error: (-1713, 'no user interaction is allowed') Googling reveals that the answer is to use pythonw, but there is no such thing installed on my system: [EMAIL PROTECTED]:~]$ pythonw -bash: pythonw: command not found Apparently, pythonw didn't get built when I installed Python 2.5, and I can't find any instructions on how to build it. (The installation instructions don't seem to mention it.) If anyone could spare a clue I would be most grateful. Thank, rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Python on a mac: how to build pythonw?
In article [EMAIL PROTECTED], Robert Kern [EMAIL PROTECTED] wrote: Ron Garret wrote: I'm trying to run the Python examples distributed with XCode and they all give me the same error: Traceback (most recent call last): File checktext.py, line 35, in module main() File checktext.py, line 8, in main pathname = EasyDialogs.AskFileForOpen(message='File to check end-of-lines in:') File /usr/local/lib/python2.5/plat-mac/EasyDialogs.py, line 650, in AskFileForOpen _interact() File /usr/local/lib/python2.5/plat-mac/EasyDialogs.py, line 53, in _interact AE.AEInteractWithUser(5000) MacOS.Error: (-1713, 'no user interaction is allowed') Googling reveals that the answer is to use pythonw, but there is no such thing installed on my system: [EMAIL PROTECTED]:~]$ pythonw -bash: pythonw: command not found Apparently, pythonw didn't get built when I installed Python 2.5, and I can't find any instructions on how to build it. (The installation instructions don't seem to mention it.) It looks like you built Python yourself. Yep. I'm a do-it-yourself kind of guy :-) The default build does not allow you to communicate with the Apple GUI. You need a framework build. I highly recommend that you simply use the binary on www.python.org instead of building from source. If you do want to build from source, please read the file Mac/README for instructions. Okeydokey. Note that in recent versions of Python, I believe that the pythonw executable is no longer necessary as a workaround. How recent? I'm already using 2.5. Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Python on a mac: how to build pythonw?
In article [EMAIL PROTECTED], Robert Kern [EMAIL PROTECTED] wrote: Ron Garret wrote: In article [EMAIL PROTECTED], Robert Kern [EMAIL PROTECTED] wrote: Note that in recent versions of Python, I believe that the pythonw executable is no longer necessary as a workaround. How recent? I'm already using 2.5. 2.5 definitely works. Yes, once you know the magic incantation: ./configure --enable-universalsdk :-) rg -- http://mail.python.org/mailman/listinfo/python-list
Python, readline and OS X
I have installed Python 2.5 on my new Intel Mac but I can't for the life of me get readline to work. I have libreadline installed, I've tried copying readline.so from my Python 2.3 installation into 2.5, I've searched the web, and no joy. Could someone please give me a clue? rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Python, readline and OS X
In article [EMAIL PROTECTED], Irmen de Jong [EMAIL PROTECTED] wrote: Ron Garret wrote: I have installed Python 2.5 on my new Intel Mac but I can't for the life of me get readline to work. I have libreadline installed, I've tried copying readline.so from my Python 2.3 installation into 2.5, I've searched the web, and no joy. Could someone please give me a clue? rg Does the info in a blog article that I wrote help? http://www.razorvine.net/frog/user/irmen/article/2006-05-08/87 No, because I'm not using Fink. But maybe I can adapt your solution. I used this when I compiled my Python 2.5 on my mac, and it seemed to work ;-) I'm now using the python.org binary distribution though and that seems to contain a working readline as well ? I'll try that too. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Python, readline and OS X
In article [EMAIL PROTECTED], James Stroud [EMAIL PROTECTED] wrote: Ron Garret wrote: I have installed Python 2.5 on my new Intel Mac but I can't for the life of me get readline to work. I have libreadline installed, I've tried copying readline.so from my Python 2.3 installation into 2.5, I've searched the web, and no joy. Could someone please give me a clue? rg Where have you installed libreadline? /usr/local/lib Is LD_LIBRARY_PATH pointing to the directory libreadline.dylib? It wasn't, but changing it so it did didn't fix the problem. (I didn't try recompiling Python, just running it. I'll try rebuilding later.) Did you install libreadline with fink? No, I just got the source from the FSF and did ./configure ; make install Bash (OSX default) and similar shells use this silly 2 part syntax: LD_LIBRARY_PATH=/sw/lib export LD_LIBRARY_PATH Actually you can do it in one line: export LD_LIBRARY_PATH=whatever :-) Do a locate libreadline.dylib and set the LD_LIBRARY_PATH to the containing directory and then make clean ./configure make make install or similar. I'll give that a whirl. Thanks. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Python, readline and OS X
In article [EMAIL PROTECTED], James Stroud [EMAIL PROTECTED] wrote: Ron Garret wrote: In article [EMAIL PROTECTED], James Stroud [EMAIL PROTECTED] wrote: Is LD_LIBRARY_PATH pointing to the directory libreadline.dylib? It wasn't, but changing it so it did didn't fix the problem. (I didn't try recompiling Python, just running it. I'll try rebuilding later.) You must re-compile python, starting with configure so that configure can identify the readline libraries. Otherwise it will compile with no readline, which is your current situation That did the trick. Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Functions, callable objects, and bound/unbound methods
If I do this:
def f(self): print self
class c1: pass
setattr(c1, 'm1', f)
Then f is automagically transmogrified into the appropriate sort of
method depending on how it is used:
c1.m1
unbound method c1.f
c1().m1
bound method c1.f of __main__.c1 instance at 0x51e738
c1().m1()
__main__.c1 instance at 0x51ec60
Note that m1 gets passed a self argument.
The same automatic transmogrification does not happen if I use an
callable instance instead of an actual function object:
class callable:
def __call__(self, *args, **kw): return args, kw
setattr(c1, 'm2', callable())
c1.m2
__main__.callable instance at 0x51e738
c1().m2
__main__.callable instance at 0x51e738
c1().m2()
((), {})
Note that no selfarg has been passed to m2.
The reason I want to do this is that I want to implement a trace
facility that traces only specific class methods. I want to say:
trace(c1.m1)
and have c1.m1 be replaced with a wrapper that prints debugging info
before actually calling the old value of m1. The reason I want that to
be an instance of a callable class instead of a function is that I need
a place to store the old value of the method so I can restore it, and I
don't want to start building a global data structure because that gets
horribly ugly, and a callable class is the Right Thing -- if there's a
way to actually make it work.
Is there? I tried the obvious things (like making callable inherit from
function, and adding im_func and im_self attribuetes) but they didn't
work.
Thanks,
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: Functions, callable objects, and bound/unbound methods
In article [EMAIL PROTECTED], Kent Johnson [EMAIL PROTECTED] wrote: Ron Garret wrote: The reason I want to do this is that I want to implement a trace facility that traces only specific class methods. I want to say: trace(c1.m1) and have c1.m1 be replaced with a wrapper that prints debugging info before actually calling the old value of m1. The reason I want that to be an instance of a callable class instead of a function is that I need a place to store the old value of the method so I can restore it, and I don't want to start building a global data structure because that gets horribly ugly, and a callable class is the Right Thing -- if there's a way to actually make it work. If the only reason for a callable class is to save a single value (the original function), you could instead store it as an attribute of the wrapper function. I considered that, and I may yet fall back on it, but 1) I wanted to understand how these things worked and 2) I need a way to tell when a method has been traced, and isinstance(method, tracer) seems less hackish to me than hasattr(method, 'saved_function'). rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Functions, callable objects, and bound/unbound methods
In article [EMAIL PROTECTED], Michele Simionato [EMAIL PROTECTED] wrote: Duncan Booth wrote: Ron Garret [EMAIL PROTECTED] wrote: I want to say: trace(c1.m1) and have c1.m1 be replaced with a wrapper that prints debugging info before actually calling the old value of m1. The reason I want that to be an instance of a callable class instead of a function is that I need a place to store the old value of the method so I can restore it, and I don't want to start building a global data structure because that gets horribly ugly, and a callable class is the Right Thing -- if there's a way to actually make it work. Is there? I tried the obvious things (like making callable inherit from function, and adding im_func and im_self attribuetes) but they didn't work. Read Functions and Methods in http://users.rcn.com/python/download/Descriptor.htm You need to implement a __get__ method on your class. See also http://groups.google.com/group/comp.lang.python/browse_frm/thread/d691240a5cfe bcdf/93503c5b9c66226e?lnk=gstq=simionato+subclassing+FunctionTypernum=1hl=e n#93503c5b9c66226e for an example and some discussion. Michele Simionato Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a reason not to do this?
In article [EMAIL PROTECTED], Carl Banks [EMAIL PROTECTED] wrote: The principle behind this is pretty much it was just a language design decision. Yes, and I'm not taking issue with the decision, just pointing out that the desire to do things differently is not necessarily perverse. P.S. If you want to be truly evil, you could use a class hook to get the modifying in-place behavior: def modify_in_place(name,bases,clsdict): cls = globals()[name] for attr,val in clsdict.iteritems(): setattr(cls,attr,val) return cls # Replace second C2 class above with this class C2: __metaclass__ = modify_in_place def m1(self): h() Doesn't work for me: c2 __main__.C2 instance at 0x51e850 c2.m1() G class C2: ... __metaclass__ = modify_in_place ... def m1(self): print 'Q' ... c2.m1() G C2().m1() Q rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a reason not to do this?
In article [EMAIL PROTECTED], Michele Simionato [EMAIL PROTECTED] wrote: Ron Garret wrote: One of the things I find annoying about Python is that when you make a change to a method definition that change is not reflected in existing instances of a class (because you're really defining a new class when you reload a class definition, not actually redefining it). So I came up with this programming style: def defmethod(cls): return lambda (func): type.__setattr__(cls, func.func_name, func) Why not just ``return lambda func: setattr(cls, func.func_name, func)`` ? Because I'm an idiot. (i.e. yes, that is obviously the right way to do it.) The only thing I don't like is that all your functions/methods will end up begin 'None'. I'd rather to be able to use the help, so I would write def defmethod(cls): def decorator(func): setattr(cls, func.func_name, func) return func return decorator @defmethod(C) def m1(self, x):pass help(m1) BTW, for people with a Lisp background I recommend using IPython with emacs and the ipython.el mode. It is pretty good, even if not comparable to Slime. Michele Simionato Good tips. Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Is there a reason not to do this?
One of the things I find annoying about Python is that when you make a change to a method definition that change is not reflected in existing instances of a class (because you're really defining a new class when you reload a class definition, not actually redefining it). So I came up with this programming style: def defmethod(cls): return lambda (func): type.__setattr__(cls, func.func_name, func) class c1(object): pass @defmethod(c1) def m1(self, x): ... Now if you redefine m1, existing instances of c1 will see the change. My question is: is there a reason not to do this? Does it screw something up behind the scenes? Is it unpythonic? Why isn't this standard operating procedure? rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a reason not to do this?
In article [EMAIL PROTECTED], Diez B. Roggisch [EMAIL PROTECTED] wrote: Ron Garret schrieb: One of the things I find annoying about Python is that when you make a change to a method definition that change is not reflected in existing instances of a class (because you're really defining a new class when you reload a class definition, not actually redefining it). So I came up with this programming style: def defmethod(cls): return lambda (func): type.__setattr__(cls, func.func_name, func) class c1(object): pass @defmethod(c1) def m1(self, x): ... Now if you redefine m1, existing instances of c1 will see the change. My question is: is there a reason not to do this? Does it screw something up behind the scenes? Is it unpythonic? Why isn't this standard operating procedure? What are you doing that needs this permanent redefinition? I like the repl, yet usually - especially when dealing with classes - I write a text file containing code. So, i just run that on a command line again, if I made some changes, recreating whatever objects I want again. Even if I'd not do that, but used a long-running interpreter inside an IDE (which is what I presume you are doing) - why do you _care_ about the old objects the first place? I mean, you obviously changed the classes for a reason. So, you are not being productive here, but still programming. Which means that you don't _have_ to care about old, unchanged objects too much. But in the end - it's your code. It will run slower, it looks kinda weird as someone who's reading it has to know what it is for, but if it suits your needs - do it. Diez I have two motivations. First, I'm dealing with some classes whose instances take a long time to construct, which makes for a long debug cycle if I have to reconstruct them after every code change. Second, the design just naturally seems to break down that way. I have a library that adds functionality (persistence) to a set of existing classes. The persistence code stores the objects in a relational DB, so it's pretty hairy code, and it has nothing to do with the functionality that the classes actually provide. The original classes are useful even with the persistence code, and I'm trying to keep things lightweight. If there's a better way to accomplish all this I'm all ears. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a reason not to do this?
In article [EMAIL PROTECTED], Hendrik van Rooyen [EMAIL PROTECTED] wrote: Ron Garret [EMAIL PROTECTED] wrote: One of the things I find annoying about Python is that when you make a change to a method definition that change is not reflected in existing instances of a class (because you're really defining a new class when you reload a class definition, not actually redefining it). So I came up with this programming style: I would have thought that not changing yesterday was the very essence of dynamism (dynamicness ??) - but that when you change something - it applies from that point in time forwards... I don't want to get into a philosophical debate. I'll just point you to CLOS as an example of an object system that already works this way. What do you propose to do about the outputs from such classes that have already happened? The ability to change methods on the fly will be used mainly for debugging I expect. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a reason not to do this?
In article [EMAIL PROTECTED], Ron Garret [EMAIL PROTECTED] wrote: In article [EMAIL PROTECTED], Hendrik van Rooyen [EMAIL PROTECTED] wrote: Ron Garret [EMAIL PROTECTED] wrote: One of the things I find annoying about Python is that when you make a change to a method definition that change is not reflected in existing instances of a class (because you're really defining a new class when you reload a class definition, not actually redefining it). So I came up with this programming style: I would have thought that not changing yesterday was the very essence of dynamism (dynamicness ??) - but that when you change something - it applies from that point in time forwards... I don't want to get into a philosophical debate. Actually, I changed my mind. Consider: def g(): print 'G' def h(): print 'H' def f(): g() class C1: def m1(self): f() class C2: def m1(self): g() c1 = C1() c2 = C2() def f(): h() class C2: def m1(self): h() c1.m1() # Prints H c2.m1() # Prints G On what principled basis can you justify two different outputs in this case? Why should I be able to change the definition of f and not have to go back and recompile all references to it, but not m1? rg -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a reason not to do this?
In article [EMAIL PROTECTED], Paul McGuire [EMAIL PROTECTED] wrote: Carl Banks [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] A straightforward, Pythonic way to do it would be to create an intermediate representation that understands both the existing class interfaces and the RDB stuff, but that could lead to synchronizing problems and a big hit in performance. And it's probably a lot of work compared to tacking on methods. OTOH, it could help with hairiness you mention. (I recently did something similar in one of my projects, though the intermediary was transient.) I would second Carl's recommendation that you find some way to persist an interim version of these expensive-to-create objects, so you can quickly load debuggable instances to accelerate your development process. With luck, you can get by with out-of-the-box marshal/unmarshal using the pickle module. We've done this several times in my office with objects that an application creates only after some extensive GUI interaction - it just slows down development too much without some quick import of debuggable instances. Even though this seems like a sidetrack, it's a pretty direct shortcut, without too much unusual technology or design work. These objects can be parts of huge networks of massively linked data structures. They are in constant flux. It is not uncommon to hit a bug after many minutes, sometimes hours, of computation. Having to store the whole shlemobble after every operation would slow things down by orders of magnitude. And writing code to be clever and only store the dirty bits would be a pain in the ass. I think I'll stick with Plan A. rg -- http://mail.python.org/mailman/listinfo/python-list
Why doesn't this work?
Python 2.3.5 (#1, Jan 30 2006, 13:30:29) [GCC 3.3 20030304 (Apple Computer, Inc. build 1819)] on darwin Type help, copyright, credits or license for more information. from datetime import datetime class ts(datetime): ... def __init__(self): pass ... ts() Traceback (most recent call last): File stdin, line 1, in ? TypeError: function takes at least 3 arguments (0 given) -- http://mail.python.org/mailman/listinfo/python-list
Re: Why doesn't this work?
In article [EMAIL PROTECTED], Larry Bates [EMAIL PROTECTED] wrote: Because datetime is a new-style class: Ah. The Constructor __new__ If you are like me, then you probably always thought of the __init__ method as the Python equivalent of what is called a constructor in C++. This isn't the whole story. When an instance of a class is created, Python first calls the __new__ method of the class. __new__ is a static method that is called with the class as its first argument. __new__ returns a new instance of the class. The __init__ method is called afterwards to initialize the instance. In some situations (think unplickling!), no initialization is performed. Also, immutable types like int and str are completely constructed by the __new__ method; their __init__ method does nothing. This way, it is impossible to circumvent immutability by explicitly calling the __init__ method after construction. I think what you wanted was: class ts(datetime): ... def __new__(self): pass ... a=ts() -Larry Thanks! rg -- http://mail.python.org/mailman/listinfo/python-list
Re: BaseHTTPServer weirdness
In article [EMAIL PROTECTED], Steve Holden [EMAIL PROTECTED] wrote: I wouldn't necessarily say you are wrong here, It's just that the cgi module has sort of just growed, so it isn't conveniently factyored for reusability in other contexts. Several people (including me) have taken a look at it with a view to possible re-engineering and backed away because of the difficulty of maintaining compatibility. Python 3K will be an ideal oppoertunity to replace it, but until then it's probably going to stay in the same rather messy but working state. It's not necessary to re-engineer cgi, just cutting and pasting and editing the code as I've done would seem to suffice. But all I'm really looking for here at this point is confirmation that I'm not in fact doing something stupid. In the past I've found that nine times out of ten if I find myself wanting to rewrite or add something to a Python module it's an indication that I'm doing something wrong. rg -- http://mail.python.org/mailman/listinfo/python-list
BaseHTTPServer weirdness
I'm trying to figure out how to use BaseHTTPServer. Here's my little
test app:
=
#!/usr/bin/python
from BaseHTTPServer import *
import cgi
class myHandler(BaseHTTPRequestHandler):
def do_GET(r):
s = ''
try:
s = cgi.parse_qs(r.rfile.read(int(r.headers.get
(Content-length))), 1)
except:
pass
r.send_response(200)
r.send_header(Content-type, text/html)
r.end_headers()
r.wfile.write(
form method=post action=foo
input type=text name=text1 value=
input type=text name=text2 value=
input type=submit
/form %s
% s)
def do_POST(r):
r.do_GET()
d = HTTPServer(('', 1024), myHandler)
d.serve_forever()
===
Two questions:
1. The line:
s = cgi.parse_qs(r.rfile.read(int(r.headers.get(Content-length))), 1)
feels like a horrible hack. It seems like this would be a better
alternative:
s = cgi.parse(r.rfile)
but that doesn't actually work. Why? What is the Right Way to parse
form data in a BaseHTTPServer?
2. Despite the fact that I'm passing a 1 for the keep_blank_values
argument to cgi.parse_qs, it doesn't actually keep blank values. Is
this a bug, or am I doing something wrong?
Thanks,
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: BaseHTTPServer weirdness
In article [EMAIL PROTECTED], Steve Holden [EMAIL PROTECTED] wrote: The normal way is s = cgi.parse() since the CGI script sees the client network socket (after consumption of HTTP headers) as its standard input. Doesn't work. (I even tried sys.stdin=r.rfile; s=cgi.parse()) Don't forget, this is not a CGI script, it's a handler for a BaseHTTPServer. 2. Despite the fact that I'm passing a 1 for the keep_blank_values argument to cgi.parse_qs, it doesn't actually keep blank values. Is this a bug, or am I doing something wrong? Sounds like a bug, but then since your parsing looks buggy I'm surprised you get anything at all. Try using a keyword argument keep_blank_values=1 just in case the order has changed or something daft. But fix your parsing first. The other thing to note is that since you are putting a dictionary's string representation out straight into your HTML if there are odd characters in it this may give you strange output in the browser, so you should view the page source to ensure that's not the case. Which it probably isn't ... I know that's not a problem because it does work when I use parse_qs. (I know about escaping HTML and all that, but this is just a little test program.) rg -- http://mail.python.org/mailman/listinfo/python-list
Re: BaseHTTPServer weirdness
In article [EMAIL PROTECTED], Steve Holden [EMAIL PROTECTED] wrote: But basically, you aren't providing a CGI environment, and that's why cgi.parse() isn't working. Clearly. So what should I be doing? Surely I'm not the first person to have this problem? I have managed to work around this for now by copying and modifying the code in cgi.parse, but this still feels like a Horrible Hack to me. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: BaseHTTPServer weirdness
In article [EMAIL PROTECTED], Steve Holden [EMAIL PROTECTED] wrote: Ron Garret wrote: In article [EMAIL PROTECTED], Steve Holden [EMAIL PROTECTED] wrote: But basically, you aren't providing a CGI environment, and that's why cgi.parse() isn't working. Clearly. So what should I be doing? Surely I'm not the first person to have this problem? I have managed to work around this for now by copying and modifying the code in cgi.parse, but this still feels like a Horrible Hack to me. Let me get this right. You are aware that CGIHTTPServer module exists. But you don't want to use that. That's right. I don't want to run CGI scripts. I don't want to launch a new process for every request. I want all requests handled in the server process. Instead you want to use your own code. No, the whole reason I'm asking this question is because I *don't* want to write my own code. It seems to me that the code to do what I want ought to be out there (or in there) somewhere and I shouldn't have to reinvent this wheel. But I can't find it. So you have ended up duplicating some of the functionality of the cgi library. And it feels like a hack. Yep. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: BaseHTTPServer weirdness
In article [EMAIL PROTECTED],
Kent Johnson [EMAIL PROTECTED] wrote:
Steve Holden wrote:
Ron Garret wrote:
In article [EMAIL PROTECTED],
Steve Holden [EMAIL PROTECTED] wrote:
But basically, you aren't providing a CGI environment, and that's why
cgi.parse() isn't working.
Clearly. So what should I be doing? Surely I'm not the first person to
have this problem?
I have managed to work around this for now by copying and modifying the
code in cgi.parse, but this still feels like a Horrible Hack to me.
Let me get this right. You are aware that CGIHTTPServer module exists.
But you don't want to use that. Instead you want to use your own code.
So you have ended up duplicating some of the functionality of the cgi
library. And it feels like a hack.
Have I missed anything? :-)
Hey, be nice. Wanting to write a request handler that actually handles a
POST request doesn't seem so unreasonable.
Except...when there are about a bazillion Python web frameworks to
choose from, why start from BaseHTTPServer? Why not use one of the
simpler frameworks like Karrigell or Snakelets or CherryPy?
It may come to that. I just thought that what I'm trying to do is so
basic that it ought to be part of the standard library. I mean, what do
people use BaseHTTPServer for if you can't parse form input?
Here is the query-handling code from Karrigell's CustomHTTPServer.py,
good at least for a second opinion:
def do_POST(self):
Begin serving a POST request. The request data must be readable
on a file-like object called self.rfile
ctype, pdict =
cgi.parse_header(self.headers.getheader('content-type'))
self.body = cgi.FieldStorage(fp=self.rfile,
headers=self.headers, environ = {'REQUEST_METHOD':'POST'},
keep_blank_values = 1, strict_parsing = 1)
# throw away additional data [see bug #427345]
while select.select([self.rfile._sock], [], [], 0)[0]:
if not self.rfile._sock.recv(1):
break
self.handle_data()
Here is CherryPy's version from CP 2.1:
# Create a copy of headerMap with lowercase keys because
# FieldStorage doesn't work otherwise
lowerHeaderMap = {}
for key, value in request.headerMap.items():
lowerHeaderMap[key.lower()] = value
# FieldStorage only recognizes POST, so fake it.
methenv = {'REQUEST_METHOD': POST}
try:
forms = _cpcgifs.FieldStorage(fp=request.rfile,
headers=lowerHeaderMap,
environ=methenv,
keep_blank_values=1)
where _cpcgifs.FieldStorage is cgi.FieldStorage with some extra accessors.
Here's what I actually ended up doing:
def parse(r):
ctype = r.headers.get('content-type')
if not ctype: return None
ctype, pdict = cgi.parse_header(ctype)
if ctype == 'multipart/form-data':
return cgi.parse_multipart(r.rfile, pdict)
elif ctype == 'application/x-www-form-urlencoded':
clength = int(r.headers.get('Content-length'))
if maxlen and clength maxlen:
raise ValueError, 'Maximum content length exceeded'
return cgi.parse_qs(r.rfile.read(clength), 1)
else:
return None
which is copied more or less directly from cgi.py. But it still seems
to me like this (or something like it) ought to be standardized in one
of the *HTTPServer.py modules.
But what do I know?
rg
--
http://mail.python.org/mailman/listinfo/python-list
Re: BaseHTTPServer weirdness
In article [EMAIL PROTECTED], Damjan [EMAIL PROTECTED] wrote: But basically, you aren't providing a CGI environment, and that's why cgi.parse() isn't working. Clearly. So what should I be doing? Probably you'll need to read the source of cgi.parse_qs (like Steve did) and see what it needs from os.environ and then provide that (either in os.environ or in a custom environ dictionary). I ended up just copying and hacking the code. It was only a dozen lines or so. But it still feels wrong. BUT why don't you use WSGI? Because BaseHTTPServer does everything I need except for this one thing. Why use a sledge hammer to squish a gnat? rg -- http://mail.python.org/mailman/listinfo/python-list
Parsing form input in a BaseHTTPServer
I'm write a web server using BaseHTTPServer. It can't be a CGI because it has to do some weird server-push stuff as database updates come in. But I still need to process form inputs as if it were a CGI. But the cgi module only works in a CGI environment. Is there something with the equivalent functionality of cgi.FieldStorage for parsing form input in a non-CGI environment such as a BaseHTTPServer? I looked at the cgi module code but it looked like it was pretty heavily intertwingled with the CGI environment variables. Thanks, rg -- http://mail.python.org/mailman/listinfo/python-list
Re: What is Expressiveness in a Computer Language
In article [EMAIL PROTECTED], Marshall [EMAIL PROTECTED] wrote: The conversation I would *really* like to have is the one where we discuss what all the differences are, functionally, between the two, and what the implications of those differences are, without trying to address which approach is right or better, because those are dependent on the problem domain anyway, and because I can make up my own mind just fine about which one I prefer. The comp.lang.functional and comp.lang.lisp people are probably two of the smartest groups on usenet. (I do not consider myself a member of either group.) You wouldn't *think* that conversation would be *so* hard to have. It's hard to have because there's very little to say, which leaves the academics without enough to do to try to justify their existence. This is the long and the short of it: 1. There are mismatches between the desired behavior of code and its actual behavior. Such mismatches are generally referred to as bugs or errors (and, occasionally, features). 2. Some of those mismatches can be detected before the program runs. Some can be detected while the program runs. And some cannot be detected until after the program has finished running. 3. The various techniques for detecting those mismatches impose varying degrees of burden upon the programmer and the user. That's it. Everything else, including but not limited to quibbling over the meaning of the word type, is nothing but postmodernist claptrap. IMHO of course. rg -- http://mail.python.org/mailman/listinfo/python-list
Re: WTF? Printing unicode strings
In article [EMAIL PROTECTED], Serge Orlov [EMAIL PROTECTED] wrote: Ron Garret wrote: I'm using an OS X terminal to ssh to a Linux machine. In theory it should work out of the box. OS X terminal should set enviromental variable LANG=en_US.utf-8, then ssh should transfer this variable to Linux and python will know that your terminal is utf-8. Unfortunately AFAIK OS X terminal doesn't set that variable and most (all?) ssh clients don't transfer it between machines. As a workaround you can set that variable on linux yourself . This should work in the command line right away: LANG=en_US.utf-8 python -c print unichr(0xbd) Or put the following line in ~/.bashrc and logout/login export LANG=en_US.utf-8 No joy. [EMAIL PROTECTED]:~$ LANG=en_US.utf-8 python -c print unichr(0xbd) Traceback (most recent call last): File string, line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in position 0: ordinal not in range(128) [EMAIL PROTECTED]:~$ What version of python and what shell do you run? What the following commands print: python -V echo $SHELL $SHELL --version [EMAIL PROTECTED]:~$ python -V Python 2.3.4 [EMAIL PROTECTED]:~$ echo $SHELL /bin/bash [EMAIL PROTECTED]:~$ $SHELL --version GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu) Copyright (C) 2002 Free Software Foundation, Inc. [EMAIL PROTECTED]:~$ -- http://mail.python.org/mailman/listinfo/python-list
Re: WTF? Printing unicode strings
In article [EMAIL PROTECTED], Serge Orlov [EMAIL PROTECTED] wrote: Ron Garret wrote: In article [EMAIL PROTECTED], Serge Orlov [EMAIL PROTECTED] wrote: Ron Garret wrote: I'm using an OS X terminal to ssh to a Linux machine. In theory it should work out of the box. OS X terminal should set enviromental variable LANG=en_US.utf-8, then ssh should transfer this variable to Linux and python will know that your terminal is utf-8. Unfortunately AFAIK OS X terminal doesn't set that variable and most (all?) ssh clients don't transfer it between machines. As a workaround you can set that variable on linux yourself . This should work in the command line right away: LANG=en_US.utf-8 python -c print unichr(0xbd) Or put the following line in ~/.bashrc and logout/login export LANG=en_US.utf-8 No joy. [EMAIL PROTECTED]:~$ LANG=en_US.utf-8 python -c print unichr(0xbd) Traceback (most recent call last): File string, line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in position 0: ordinal not in range(128) [EMAIL PROTECTED]:~$ What version of python and what shell do you run? What the following commands print: python -V echo $SHELL $SHELL --version [EMAIL PROTECTED]:~$ python -V Python 2.3.4 [EMAIL PROTECTED]:~$ echo $SHELL /bin/bash [EMAIL PROTECTED]:~$ $SHELL --version GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu) Copyright (C) 2002 Free Software Foundation, Inc. [EMAIL PROTECTED]:~$ That's recent enough. I guess the distribution you're using set LC_* variables for no good reason. Nope: [EMAIL PROTECTED]:~$ export | grep LC [EMAIL PROTECTED]:~$ Either unset all enviromental variables starting with LC_ and set LANG variable or overide LC_CTYPE variable: LC_CTYPE=en_US.utf-8 python -c print unichr(0xbd) Should be working now :) Nope: [EMAIL PROTECTED]:~$ LC_CTYPE=en_US.utf-8 python -c print unichr(0xbd) Traceback (most recent call last): File string, line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in position 0: ordinal not in range(128) rg -- http://mail.python.org/mailman/listinfo/python-list
