Re: Python 2.7.3, C++ embed memory leak?
Qi n...@no.com writes: Hi guys, Is there any known memory leak problems, when embed Python 2.7.3 in C++? I Googled but only found some old posts. I tried to only call Py_Initialize() and Py_Finalize(), nothing else between those functions, Valgrind still reports memory leaks on Ubuntu? Is that a know problem? Did Python 3.x solve it? I want some confirmation. Python does some special things that confuse valgrind. Don't bother. http://svn.python.org/projects/python/trunk/Misc/README.valgrind Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: ctypes callback with char array
ohlfsen ohlf...@gmail.com writes: Hello. Hoping that someone can shed some light on a tiny challenge of mine. Through ctypes I'm calling a c DLL which requires me to implement a callback in Python/ctypes. The signature of the callback is something like void foo(int NoOfElements, char Elements[][100]) How do I possible implement/process char Elements[][100] in Python/ctypes code? I'd try it like this: $ python Python 2.7 (r27:82500, May 2 2011, 22:50:11) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type help, copyright, credits or license for more information. oWelcome to rlcompleter2 0.98 for nice experiences hit tab multiple times from ctypes import * callback_type = CFUNCTYPE(None, c_int, POINTER(c_char_p)) def my_callback(int, elements): ... pass ... c_callback = callback_type(my_callback) c_callback CFunctionType object at 0x100697940 No need to know that it's a pointer to char[100] pointers - you can cast that if you want to. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: pypi and dependencies
Andrea Crotti andrea.crott...@gmail.com writes: When I publish something on Pypi, is there a way to make it fetch the list of dependencies needed by my project automatically? It would be nice to have it in the Pypi page, without having to look at the actual code.. Any other possible solution? I don't understand this - if you declare the dependencies of your published package in the requires-section of the setup()-call, easy_install (and I guess pip as well, no experience with it) will automatically fetch these. Is that what you wanted to know? Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: configobj validation
Andrea Crotti andrea.crott...@gmail.com writes:
On 03/19/2012 12:59 PM, Andrea Crotti wrote:
I seemed to remember that type validation and type conversion worked
out of the box, but now
I can't get it working anymore.
Shouldn't this simple example actually fail the parsing (instead it
parses perfectly port to a string)?
sample.py:
from configobj import ConfigObj
conf = ConfigObj('sample.conf', configspec='sample.spec')
sample.conf:
port = some_string
sample.spec:
port = integer(0, 10)
PS. using configobj 4.7.2
I now checked the repo and configobj seems also quite dead (2 years
ago last version), it's a pity because it's a really nice library.
Any other alternatives for validation/configuration systems otherwise?
It works - so why do you bother? And I'm not sure about the above code -
AFAIK, validation is a two-step thing:
http://www.voidspace.org.uk/python/articles/configobj.shtml#validation
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: Decoding unicode is not supported in unusual situation
John Nagle na...@animats.com writes: I think that somewhere in suds, they subclass the unicode type. That's almost too cute. The proper test is isinstance(s,unicode) Woot, you finally discovered polymorphism - congratulations! Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: A http server
Back9 backgoo...@gmail.com writes: Hi, I'm trying to set up a http server to handle a single POST request. That POST request is to upload a huge file and the server is supposed to handle it with the just POST request. With my python sample code, multiple post requests are working well, but that is not my solution. I need a single POST request handling in the http server side. Does anyone have a good idea for this case or sample code? Why doesn't the single post not work with your sample code? Where is your sample code? It might be a good idea to use WSGI for this, to stream out the POST-body to a temporary file. Can you use mod_wsgi in conjuction with Apache? If not, some pure WSGI-server implementation such as Paster might be enough. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there anyway to run JavaScript in python?
crow wen...@gmail.com writes: Hi, I'm writing a test tool to simulate Web browser. Is there anyway to run JavaScript in python? Thanks in advance. Not really. Yes, you can invoke spidermonkey. But the crucial point about running JS is not executing JS, it's about having the *DOM* of the browser available. Which spidermonkey obviously hasn't. So, I recommend using Selenium. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Interrput a thread
gervaz ger...@gmail.com writes: On 31 Dic 2010, 23:25, Alice Bevan–McGregor al...@gothcandy.com wrote: On 2010-12-31 10:28:26 -0800, John Nagle said: Even worse, sending control-C to a multi-thread program is unreliable in CPython. See http://blip.tv/file/2232410; for why. It's painful. AFIK, that has been resolved in Python 3.2 with the introduction of an intelligent thread scheduler as part of the GIL release/acquire process. - Alice. Ok, but then suppose I have multiple long running threads that I want to delete/suspend because they are tooking too much time, which solution do you propose? If possible, use multiple processes instead. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: CPython on the Web
azakai alonmozi...@gmail.com writes: Hello, I hope this will be interesting to people here: CPython running on the web, http://syntensity.com/static/python.html That isn't a new implementation of Python, but rather CPython 2.7.1, compiled from C to JavaScript using Emscripten and LLVM. For more details on the conversion process, see http://emscripten.org A fun hack. Have you bothered to compare it to the PyPy javascript backend - perfomance-wise, that is? Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: CPython on the Web
Gerry Reno gr...@verizon.net writes: On 01/03/2011 03:13 PM, Diez B. Roggisch wrote: A fun hack. Have you bothered to compare it to the PyPy javascript backend - perfomance-wise, that is? Diez I don't think that exists anymore. Didn't that get removed from PyPy about 2 years ago? Ah, didn't know that. I was under the impression pyjamas was done with it. Apparently, that's wrong: http://pyjs.org/ But then I re-phrase my question: how does this relate to pyjamas/pyjs? Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Interrput a thread
gervaz ger...@gmail.com writes: On 3 Gen, 22:17, Adam Skutt ask...@gmail.com wrote: On Jan 3, 4:06 pm, Jean-Paul Calderone calderone.jeanp...@gmail.com wrote: Multiple processes, ok, but then regarding processes' interruption there will be the same problems pointed out by using threads? No. Processes can be terminated easily on all major platforms. See `os.kill`. Yes, but that's not the whole story, now is it? It's certainly much more reliable and easier to kill a process. It's not any easier to do it and retain defined behavior, depending on exactly what you're doing. For example, if you kill it while it's in the middle of updating shared memory, you can potentially incur undefined behavior on the part of any process that can also access shared memory. In short, taking a program that uses threads and shared state and simply replacing the threads with processes will likely not gain you a thing. It entirely depends on what those threads are doing and how they do it. Adam As per the py3.1 documentation, os.kill is only available in the Unix os. Regarding the case pointed out by Adam I think the best way to deal with it is to create a critical section so that the shared memory will be updated in an atomic fashion. Btw it would be useful to take a look at some actual code/documentation in order to understand how others dealt with the problem... There is the multiprocessing module. It's a good start, and works cross-platform. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: position independent build of python
erikj tw55...@gmail.com writes: If my understanding is correct, the sys.prefix variable holds the root directory python uses to find related files, and eg its site-packages. the value of sys.prefix is specified at compile time. it seems that on windows, when I build/install python at one location, and later move it to another location, python will find all its needs relative to the new location. however, when I do the same on linux, python keeps looking for its dependencies in the build location. is there a possibility to have it always look at a position relative to the location of the executable ? the goal is to be able to build python on one machine, and then simply copy the build tree to other machines at other locations. Maybe looking at virtualenv, especially with the --no-site-packages variable set gives you a hint. AFAIK there are some hard-coded paths involved, but not inside compiled code. Maybe you can remedy that somehow. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: pyqt4: multi-threaded database access
Adrian Casey caseyal...@adam.com.au writes: I have a PyQt4 multi-threaded application which accesses many hosts concurrently via ssh. I would like each thread to have access to a database so that it can look up details about the particular system it is connected to. The easy way is to have each thread create a connection to the database. However, this is wasteful and likely to exhaust the maximum number of connections the database (postgresql) allows. Instead, I would like to share a single database connection between all threads. In PyQt4, this is not possible since the database connection can only be used by the thread which created it. So, I'm thinking I'll create a thread to just handle database queries. The worker threads will communicate with the database thread using signals and slots. My question is, if I have 20 worker threads who have emitted a signal which results in a database query, how do I ensure that the query results are sent back to the originating thread? Ideas, examples anyone? First of all, do you *know* it will exhaust the connections, or is that just some gut feeling? Because postgres can easily handle a few hundered connections, and unless you really are exhausting these, it's the easiest solution. The next advice would be to create a pool of connections (pure python, not Qt). Then you lose the can only be used by the thread which created it restriction. Of course you should only use a connection from within one thread at a time, but that shouldn't be an issue. So all you have to make sure is that you fetch return the connections properly. I would use a context-manager for that: with connection_pooll as connection: do_some_queries(connection) HTH, Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: How to correctly pass “pointer-to-pointer ” into DLL via ctypes?
Grigory Petrov grigory@gmail.com writes:
Hello.
I have a DLL that allocates memory and returns it. Function in DLL is like
this:
void Foo( unsigned char** ppMem, int* pSize )
{
* pSize = 4;
* ppMem = malloc( * pSize );
for( int i = 0; i * pSize; i ++ ) (* pMem)[ i ] = i;
}
Also, i have a python code that access this function from my DLL:
from ctypes import *
Foo = windll.mydll.Foo
Foo.argtypes = [ POINTER( POINTER( c_ubyte ) ), POINTER( c_int ) ]
mem = POINTER( c_ubyte )()
size = c_int( 0 )
Foo( byref( mem ), byref( size ) ]
print size, mem[ 0 ], mem[ 1 ], mem[ 2 ], mem[ 3 ]
I'm expecting that print will show 4 0 1 2 3 but it shows 4 221 221
221 221 O_O. Any hints what i'm doing wrong?
After correcting quite a few obvious errors in your code, it worked just
fine for me. So I guess what you think you test is not what you test.
--- test.py
from ctypes import *
foo =
cdll.LoadLibrary(/Users/deets/projects/GH28/kinect/kinect/c/foo/libtest.dylib).foo
foo.argtypes = [ POINTER( POINTER( c_ubyte ) ), POINTER( c_int ) ]
mem = POINTER( c_ubyte )()
size = c_int( 0 )
foo( byref( mem ), byref( size ) )
print size, mem[ 0 ], mem[ 1 ], mem[ 2 ], mem[ 3 ]
---
--- test.c
void foo( unsigned char** ppMem, int* pSize )
{
int i;
* pSize = 4;
* ppMem = malloc( * pSize );
for( i = 0; i * pSize; i ++ ) (* ppMem)[ i ] = i;
}
---
--- CMakeLists.txt
cmake_minimum_required (VERSION 2.6)
project (Test)
add_library(test SHARED test.c)
---
--
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: Round Trip: C to Python to C Module
Eric Frederich eric.freder...@gmail.com writes: I have a proprietary software PropSoft that I need to extend. They support extensions written in C that can link against PropLib to interact with the system. I have a Python C module that wraps a couple PropLib functions that I call PyProp. From an interactive Python shell I can import PyProp and call a function. None of these functions really do anything outside the context of being logged into the PropSoft software; so all the functions fail when running from Python alone. To my amazement, I was able to run PyRun_SimpleString(import PyProp\nPyProp.some_function()) without setting PYTHONPATH or anything. How this works, I don't know and I don't really care (at the moment anyway). The problem I'm having now is how do I return things from my Python script back to C? Ultimately I won't be hard coding python inside of PyRun_SimpleString but loading the script from a file. So, how do I return values back to C? Python functions return values but running a python script?... doesn't that just have an exit status? Is there a mechanism for doing this? You write an extension in C that embeds a Python-interpreter. That interpreter then loads your script, and executes it. Let's say the PropSoft offers you three functions for a custom extension. void init(); int do_something(int); void cleanup(); Then you write a C-extension that contains these three function calls, and in init, you create a Python-interpreter and load your script. It should offer a do_something_in_python method that takes an int and returns one. In do_something, you invoke the aforementioned do_something_in_python function through the Python C-API as described. In cleanup, you do what the function name says. Possibly the use of elmer[1] makes the whole process easier. [1]: http://wiki.python.org/moin/elmer -- Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: QT window closes immediately
Martin Caum herrxan...@gmail.com writes:
I am attempting to open a window on mouse activity which works, but
the window fails to stay open.
I set it to terminate when the escape key is pressed even when the
program is not currently selected. This works fine. Originally I had
it create the window only with a right click, but when I noticed the
window closed immediately I set it to any mouse activity for easier
debugging. My script is as follows:
[code]
import pythoncom, pyHook
import sys
import win32api
from PyQt4 import QtGui, QtCore
class WindowObject(QtGui.QWidget):
def __init__(self, parent=None):
QtGui.QWidget.__init__(self, parent)
self.setWindowTitle('Window')
self.resize(50, 250)
def OnKeyboardEvent(event):
if event.KeyID == 27:
win32api.PostQuitMessage()
return 1
def OnMouseEvent(event):
x = event.Position[0]
y = event.Position[1]
createWindow(x, y)
return 1
def createWindow(x, y):
menu = WindowObject()
menu.move(x, y)
menu.show()
hm = pyHook.HookManager()
hm.MouseAll = OnMouseEvent
hm.KeyDown = OnKeyboardEvent
hm.HookMouse()
hm.HookKeyboard()
app = QtGui.QApplication(sys.argv)
pythoncom.PumpMessages()[/code]
I'm fairly certain that this is due to my lack of understanding in the
PumpMessages command and the HookManager. Any advice?
1) Keep a reference to your window-object. I can only guess (PyQt not
being a regular topic when hacking for me.. .sadly), but GC can
interfere with these kind of things.
2) what is the win32 stuff about? Qt should abstract from that.
3) if in doubt, as on the PyQt mailing list.
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: Getting references to objects without incrementing reference counters
Artur Siekielski artur.siekiel...@gmail.com writes: On Nov 15, 1:03 am, de...@web.de (Diez B. Roggisch) wrote: You don't say what data you share, and if all of it is needed for each child. So it's hard to suggest optimizations. Here is an example of such a problem I'm dealing with now: I'm building large index of words in memory, it takes 50% of RAM. After building it I want to enable querying it using multiple processors enabling fast handling of multiple clients at once (communication is done with sockets). So I create the index in the parent process and than fork children that handle client queries. But after a short time only traversing the index by children makes them use much memory, resulting shortly in Out Of Memory Errors. And AFAIK there is no built-in way of doing what you want. It's complex and error-prone. Having read-only objects is complex and error-prone? Yes. Python has no built-in semantics for that. You can't declare an object to be read-only. And if one wanted to implement that, you'd have to make sure the object consists only of read-only attributes itself. And circumvene a great deal of the dynamic features in python (which you don't need for this usecase, but still are there) Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Why datetime module is so complicated?
Zeynel azeyn...@gmail.com writes: It's about a week now I've been trying to convert a datetime object to seconds since epoch; the object is set to current time by class Rep() in Google App Engine: class Rep(db.Model): ... mCOUNT = db.IntegerProperty() mDATE0 = db.DateTimeProperty(auto_now_add=True) mWEIGHT = db.FloatProperty() I want to divide mDATE0 by the integer mCOUNT. I asked the same question here previously and also at stackoverflow. So far, I still cannot make it work. I would greatly appreciate if someone who is an expert in datetime operation in Python could help me understand this issue. Thank you. from datetime import * d = datetime.now() dir(d) ['__add__', '__class__', '__delattr__', '__doc__', '__eq__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__ne__', '__new__', '__radd__', '__reduce__', '__reduce_ex__', '__repr__', '__rsub__', '__setattr__', '__str__', '__sub__', 'astimezone', 'combine', 'ctime', 'date', 'day', 'dst', 'fromordinal', 'fromtimestamp', 'hour', 'isocalendar', 'isoformat', 'isoweekday', 'max', 'microsecond', 'min', 'minute', 'month', 'now', 'replace', 'resolution', 'second', 'strftime', 'strptime', 'time', 'timetuple', 'timetz', 'today', 'toordinal', 'tzinfo', 'tzname', 'utcfromtimestamp', 'utcnow', 'utcoffset', 'utctimetuple', 'weekday', 'year'] d.time() datetime.time(17, 50, 54, 778159) d.ctime() 'Sun Nov 14 17:50:54 2010' d.time built-in method time of datetime.datetime object at 0x1003210f8 d.timetuple() (2010, 11, 14, 17, 50, 54, 6, 318, -1) import time dir(time) ['__doc__', '__file__', '__name__', 'accept2dyear', 'altzone', 'asctime', 'clock', 'ctime', 'daylight', 'gmtime', 'localtime', 'mktime', 'sleep', 'strftime', 'strptime', 'struct_time', 'time', 'timezone', 'tzname', 'tzset'] help(time.mktime) time.mktime(d.timetuple()) 1289753454.0 Not that hard. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Getting references to objects without incrementing reference counters
Artur Siekielski artur.siekiel...@gmail.com writes: Hi. I'm using CPython 2.7 and Linux. In order to make parallel computations on a large list of objects I want to use multiple processes (by using multiprocessing module). In the first step I fill the list with objects and then I fork() my worker processes that do the job. This should work optimally in the aspect of memory usage because Linux implements copy-on-write in forked processes. So I should have only one physical list of objects (the worker processes don't change the objects on the list). The problem is that after a short time children processes are using more and more memory (they don't create new objects - they only read objects from the list and write computation result to the database). After investigation I concluded the source of this must be incrementing of a reference counter when getting an object from the list. It changes only one int but OS must copy the whole memory page to the child process. I reimplemented the function for getting the element (from the file listobject.c) but omitting the PY_INCREF call and it solved my problems with increasing memory. The questions is: are there any better ways to have a real read-only list (in terms of memory representation of objects)? My solution is of course not safe. I thought about weakrefs but it seems they cannot be used here because getting a real reference from a weakref increases a reference counter. Maybe another option would be to store reference counters not in objects, but in a separate array to minimize number of memory pages they occupy... You don't say what data you share, and if all of it is needed for each child. So it's hard to suggest optimizations. And AFAIK there is no built-in way of doing what you want. It's complex and error-prone. Maybe mmap + (struct|pickle) help, if what you need can be formulated in a way that traversing the whole data piecewise by explicitly marshaling-demarshaling data? Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: strange behavor....
alex23 wuwe...@gmail.com writes: Tracubik affdfsdfds...@b.com wrote: why the integer value doesn't change while the list value do? http://effbot.org/pyfaq/why-are-default-values-shared-between-objects.htm Not the issue here. The reason the OP sees a difference that there is only one way to pass parameters in python. There have been wars waged about the right name to call them. What's happening is that in the function, the parameter names are bound to the objects the caller passed in. But these names only exist in the scope of the function. So re-binding that name by doing e.g. a = 2 does not change the binding in the callers context. So because of this, no change happens in change_integer. Because there is no change to the integer itself. Actually, you can't even change an integer in pyhon. They are immutable. a = 2 a += 10 will make a point to the integer-object with the value 12. But the old 2 and 10 still exist. And that's where the differency in change_list is in. That gets passed a reference to a _mutable_ object, a list. And if you mutatet that list, you end up with a changed object in the callers context as well. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: scipy code runs in empty directory, not another
Beliavsky beliav...@aol.com writes: After installing numpy, scipy, and matplotlib for python 2.6 and running the code from http://www.scipy.org/Cookbook/OptimizationDemo1 (stored as xoptimize.py) in a directory with other python codes, I got the error messages C:\python\code\mycodepython xoptimize.py Traceback (most recent call last): File xoptimize.py, line 3, in module from pylab import * File c:\python26\lib\site-packages\pylab.py, line 1, in module from matplotlib.pylab import * File c:\python26\lib\site-packages\matplotlib\__init__.py, line 133, in module import sys, os, tempfile File c:\python26\lib\tempfile.py, line 34, in module from random import Random as _Random File C:\python\code\mycode\random.py, line 1, in module from RandomArray import standard_normal File C:\python\code\mycode\RandomArray.py, line 1, in module import ranlib ImportError: No module named ranlib When I create a new directory, copy xoptimize.py there, and run, the program works. Can someone explain why other files are interfering in the first case? Thanks. Because you probably have a file named the same way as some pyhon or matplotlib module, and that shadows the real one. Try copying more and more files into your new directory to find the culprit. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: How find all childrens values of a nested dictionary, fast!
macm moura.ma...@gmail.com writes:
Hi Folks
How find all childrens values of a nested dictionary, fast!
There is no faster than O(n) here.
a = {'a' : {'b' :{'/' :[1,2,3,4], 'ba' :{'/' :[41,42,44]} ,'bc'
:{'/':[51,52,54], 'bcd' :{'/':[68,69,66]}}},'c' :{'/' :[5,6,7,8]}}, 'ab' :
{'/' :[12,13,14,15]}, 'ac' :{'/' :[21,22,23]}}
a['a']
{'c': {'/': [5, 6, 7, 8]}, 'b': {'ba': {'/': [41, 42, 44]}, '/': [1,
2, 3, 4], 'bc': {'bcd': {'/': [68, 69, 66]}, '/': [51, 52, 54]}}}
a['a']['b']
{'ba': {'/': [41, 42, 44]}, '/': [1, 2, 3, 4], 'bc': {'bcd': {'/':
[68, 69, 66]}, '/': [51, 52, 54]}}
a['a']['b'].values()
[{'/': [41, 42, 44]}, [1, 2, 3, 4], {'bcd': {'/': [68, 69, 66]}, '/':
[51, 52, 54]}]
Now I want find all values of key /
Desire result is [41, 42, 44, 1, 2, 3, 4, 68, 69, 66, 51, 52, 54]
a = {'a' : {'b' :{'/' :[1,2,3,4], 'ba' :{'/' :[41,42,44]} ,'bc'
:{'/':[51,52,54], 'bcd' :{'/':[68,69,66]}}},'c' :{'/' :[5,6,7,8]}}, 'ab' : {'/'
:[12,13,14,15]}, 'ac' :{'/' :[21,22,23]}}
def f(d):
if / in d:
for v in d[/]:
yield v
for value in d.values():
if isinstance(value, dict):
for v in f(value):
yield v
print list(f(a))
But that gives me a different result:
[5, 6, 7, 8, 1, 2, 3, 4, 41, 42, 44, 51, 52, 54, 68, 69, 66, 21, 22,
23, 12, 13, 14, 15]
Why don't you expect e.g. 23 in your output?
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: no line breaks in xml file with elementTree
hackingKK hackin...@gmail.com writes: On Sunday 31 October 2010 01:58 PM, Lawrence D'Oliveiro wrote: In messagemailman.412.1288509358.2218.python-l...@python.org, hackingKK wrote: I want to know if there is a way to have the ElementTree module write to an xml file with line breaks? Why does it matter? The XML files you generate are not for humans to look at, are they? So is there a function to generate tags with namespace? http://lmgtfy.com/?q=element+tree+namespac -- Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: xml : remove a node with dom
alain walter alain.serge.wal...@gmail.com writes: Hello, I have many difficulties to manipulate xml routines. I'm working with python 2.4.4 and I cannot change to a more recent one, then I use dom package, why not. In the following code, I'm trying unsuccessfully to remove a particular node. It seems to me that it should be basic, but it's not. Thanks for your help If you post code, make an attempt for it to be runnable. I had to fix numerous simple errors to make it run actually solve your problem. from xml.dom.minidom import parse,parseString from xml.dom import Node toxml= aixm:VORTimeSlice xmlns:aixm=aixm xmlns:gml=gml gml:id=ABB aixm:typeABB/aixm:type aixm:designatorABB/aixm:designator aixm:ElevatedPoint gml:id=ABB srsDimension=2 gml:pos srsDimension=2-51.23 4.6501/gml:pos aixm:elevation uom=Mxxx_toremove_xxx/aixm:elevation /aixm:ElevatedPoint /aixm:VORTimeSlice dom = parseString(toxml) def ApplicationWhitespaceRemoving(ele) : for c in ele.childNodes: if c.nodeType == c.TEXT_NODE: if c.nodeValue == xxx_toremove_xxx: parent = c.parentNode parent.removeChild(c) elif c.nodeType == ele.ELEMENT_NODE: ApplicationWhitespaceRemoving(c) ApplicationWhitespaceRemoving(dom) print dom.toxml() -- Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: downcasting problem
Nikola Skoric n...@fly.srk.fer.hr writes: Hi everybody, I need to downcast an object, and I've read repeatedly that if you need to downcast, you did something wrong in the design phase. So, instead of asking how do you downcast in python, let me explain my situation. I have a 2-pass parser. 1st pass ends up with a bunch of superclass object, and 2nd pass is supposed to downcast those to one of subclasses so I can work with them. This is usually called a 'reducer' in parsing, and it's purpose is to rewrite the parsing result to something new. It's usually implemented using a visitor-pattern. In your case, it should be rather straightforward, by simply iterating over the objects and creating new, more specialized types depending on what you actually need. So you don't need to delegate anything. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Getting returncode of a command executed with Popen through xterm
amfr...@web.de writes: Hi, i have a program that have to execute linux commands. I do it like this: retcode = Popen([xterm, -e, command],stdin=PIPE, stdout=PIPE, stderr=PIPE) I have to use xterm because some commands need further input from the user after they are executed. But when i use xterm i can't get the returncode or the errormessage from a command: print retcode.returncode # always 0 print retcode.stderr.read() # always empty print retcode.stdout.read() # always empty The same code works without xterm. As i understand it, if i use xterm the retcode refers to the xterm window (process). But is there a way i can get the returncode and errormessage of the command i sent to xterm ? You could create a python-wrapper-script that will store the result and streams in files. Like this command = [callwrapper, --dest-key=random_string, the_real_command] Popen([xterm, -e, command]) The dest-key will be used to create files named random_string.status, random_string.stdout, random_string.stderr so that you can read from them afterwards. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: error in loading data into the code
nakisa nakisa.noor...@gmail.com writes: hello , I have started python last week,so maybe my question is a bit stupid. I got this error in my simple python code while trying to load data into the code. I have added this libaraies from pylab import * from scipy import * import matplotlib.mlab as mlab from numpy import * from scipy import optimize but get this following error : /usr/lib/pymodules/python2.6/matplotlib/mlab.py:1267: DeprecationWarning: use numpy.loadtxt warnings.warn(use numpy.loadtxt, DeprecationWarning) # 2009/06/13 any idea ? You don't get an error. You get a warning. Crucial difference there. You can ignore it. You can even suppress it. But you shouldn't bother, the authors of mlab should. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: python/c api
alex23 wuwe...@gmail.com writes: On Oct 15, 5:53 am, de...@web.de (Diez B. Roggisch) wrote: For example Ableton Live, an audio sequencer. I _have_ Live and I didn't realise this :O Thanks! Well, it's not a feature for end-users, it's used internally for some midi controller mapping stuff. Our API so to say is Max4Live. But Python is used, through Boost::Python. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: send command to parent shell
Martin Landa landa.mar...@gmail.com writes: Hi, is there a way how to send command from python script to the shell (known id) from which the python script has been called? More precisely, the goal is to exit running bash (on Linux) or cmd (on Windows) directly from wxPython application, currently user needs to quit wxPython application and then underlaying command prompt by 'exit' command. Why is it started from the shell then in the first place? And if I did it, I beg you to *not* close it... if I wanted that, I would have used exec program Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: python/c api
Tony tonywinslow1...@gmail.com writes: hi, is the python/c api extensively used? and what world-famous software use it? thanks! It is, for a lot of extensions for python, and a lot of embedding python into a software. For example Ableton Live, an audio sequencer. Arc GIS has it, and the Eve Online. Many more do, I guess. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: GCC process not working as expected when called in Python (3.1.2) subprocess-shell, but OK otherwise
Kingsley Turner kingsley.tur...@openfieldcommunications.com writes: Hi, I'm using GCC as a pre-processor for a C-like language (EDDL) to handle all the includes, macros, etc. producing a single source file for another compiler. My python code massages the inputs (which arrive in a .zip file), then calls GCC. I have a problem where if I call GCC from my python script some of the #defines are not processed in the output. However if I paste the exact same GCC command-line into a shell, I get a correct output. I'm calling GCC in this manner: ### Execute GCC, keep stdout stderr err_out = open(error_filename,wb) process = subprocess.Popen(gcc_command, stderr=err_out, bufsize=81920, cwd=global_options['tmp']) gcc_exit_code = process.wait() log(GCC Exit Code %u % (gcc_exit_code)) err_out.close() where gcc_command is: /usr/bin/gcc -I /tmp/dd-compile_1286930109.99 -I /home/foo/eddl-includes -D__TOKVER__=600 -ansi -nostdinc -v -x c -E -o /tmp/dd-compile_1286930109.99/11130201.ddl.OUT /tmp/dd-compile_1286930109.99/11130201.ddl So when this code spawns GCC, the compiler does not really work 100%, but if I paste this exact command line, the output is perfect. I'm not really sure how to debug this. I already checked the ulimits, and permissions shouldn't be a problem since it's all run by the same user, I also checked the environment - these were copied into the subshell. GCC produces no warnings, or errors. The output is mostly OK, some other macros have been processed. If I diff the working output with the non-working one, the differences are only a bunch of skipped #defines. gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5) Python 3.1.2 (release31-maint, Sep 17 2010, 20:27:33) Linux 2.6.35-22-generic #34-Ubuntu SMP Sun Oct 10 09:26:05 UTC 2010 x86_64 GNU/Linux Any suggestions for helping me debug this would be much appreciated. sounds nasty. Only thing I can imagine is that GCC wants specific environment variables to exist. Maybe using shell=True helps? Or capturing and passing an explicit environment? Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Compiling as 32bit on MacOSX
Gregory Ewing greg.ew...@canterbury.ac.nz writes: Philip Semanchuk wrote: Hi Greg, Are you talking about compiling Python itself or extensions? I've managed to get Python itself compiled as 32 bit, and that also seems to take care of extensions built using 'python setup.py ...'. I'm mainly concerned about non-Python libraries that get wrapped by the extensions, of which I've built up quite a collection over the years. Currently I'm having to keep a careful eye out when building them to make sure they don't get compiled with the wrong architecture, since gcc's natural inclination is to default to 64 bit whenever it's available. So I was wondering if there was some way of globally changing that default that doesn't rely on compiler options getting passed correctly through the many and varied layers of build technology that one comes across. But from what I've seen so far, it seems not. If all you have is a fat-binary, you can still work with that using the lipo-tool to remove those architectures you are not interested in. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Difficulty in easy_install
John Nagle na...@animats.com writes: On 10/11/2010 1:45 AM, sankalp srivastava wrote: I am having difficulty in easy_installing I use a proxy server and strange errors , like it can't fetch the package is showing up . the package is pyspeech ...please help me :( I don't know if the proxy server is causing the problems , in linux i would have import http-proxy in .bashrc but am not a windows expert can anyone help me plz That's about typical. easy_install isn't easy. It has some built-in assumptions about where various files are stored, and if the installation doesn't match them, installs fail with obscure error messages. I've been using easy_install and eggs for years, and at least since the advent ogf virtualenve they haven't given me any troubles whatsoever. So I guess it's an OSI layer 8 issue. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: how to add patch
jimgardener jimgarde...@gmail.com writes: hi I have some demo python code hosted on a public host that uses subversion..and I want to modify one of the files using a patch file handed to me by another person..How do I do this?Generally I checkout the code and make the change and then commit again..I have never done through patch..Can somebody tell me how to do this? This is not really python specific. Which is not to say that you are wrong here asking, but there are a bazillion of sources out there to help you. Google. Read man-pages. Try it out. My first attempt would be cd my_project patch -p0 the.patch But it might not work, depending on how the patch was created. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: emdding python gui in c code - OS independent
tinauser tinau...@libero.it writes: On Oct 11, 6:49 pm, Chris Rebert c...@rebertia.com wrote: On Mon, Oct 11, 2010 at 6:16 AM, tinauser tinau...@libero.it wrote: hi there, i need to embed python GUI in a c++ code. I've seen that,while on windows running GUI is no problem, in mac i need to use pythonw instead python. the question is,how should i tell the program that if the OS is mac, it should pythonw, otherwise python is fine? I think you have it backwards. MS Windows is where one typically needs to use pythonw to suppress a console window from opening when Python is run. *nixes (including Mac OS X) have no such problem and (I'm pretty sure) only have a pythonw executable for compatibility purposes. Just specify pythonw regardless of OS and you should be fine. This is not entirely true, there is a difference on the mac, see below. Cheers, Chris --http://blog.rebertia.com thanks: how do i specify pythonw programmatically? i tried Py_SetProgramName(pythonw); it doesn't raise errors, but does not solve the problem on mac, i.e. i get the error: This program needs access to the screen. Please run with 'pythonw', not 'python', and only when you are logged in on the main display of your Mac. I'm not sure that's your problem. It is not about the *name* of the program. It is about Python being a Framework build. Which the ones that come with OSX are, and if you link against them that's enough. However, you are writing your *own* binary program, and I guess that's the problem. I don't know without further research why there is a difference between commandline and GUI-programs in OSX (I guess it has to do with the event loop or something), but I think your umbrella program is the real culprit here. If it's not a proper GUI program (whatever that exactly means), delegating rendering of a GUI to Python will trigger that code that spits out your above error message. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Difficulty in easy_install
sankalp srivastava richi18...@gmail.com writes: I am having difficulty in easy_installing I use a proxy server and strange errors , like it can't fetch the package is showing up . the package is pyspeech ...please help me :( I don't know if the proxy server is causing the problems , in linux i would have import http-proxy in .bashrc but am not a windows expert can anyone help me plz Please show us the full output of easy-install. And can you install other packages? Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: how to handle network failures
harryos oswald.ha...@gmail.com writes: hi I am trying to write a DataGrabber which reads some data from given url..I made DataGrabber as a Thread and want to wait for some interval of time in case there is a network failure that prevents read(). I am not very sure how to implement this class DataGrabber(threading.Thread): def __init__(self,url): threading.Thread.__init__(self) self.url=url def run(self): data=self.get_page_data() process_data(data) def get_page_data(): try: f=urllib.urlopen(self.url) data=f.read(1024) except IOError: #wait for some time and try again time.sleep(120) data=self.get_page_data() return data Is this the way to implement the part where the thread waits and reads the data again? Will this handle network failures?Can somebody please help? This only works if your page is always 1024 bytes long. Which I doubt. So don't pass the 1024 to read. Also, you need a loop to re-read the data. Like this: for n in xrange(max_number_of_retries): try: f=urllib.urlopen(self.url) data = f.read() break # exist the loop if all except IOError: pass self.process_data(data) Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: how to find difference in number of characters
harryos oswald.ha...@gmail.com writes: On Oct 9, 4:52 pm, Peter Otten __pete...@web.de wrote: You might get more/better answers if you tell us more about the context of the problem and add some details that may be relevant. Peter I am trying to determine if a wep page is updated by x number of characters..Mozilla firefox plugin 'update scanner' has a similar functionality ..A user can specify the x ..I think this would be done by reading from the same url at two different times and finding the change in body text..I was wondering if difflib could offer something in the way of determining the size of delta.. If you normalize the data, this might be worth trying. Make all tags appear on one single line, possibly re-order attributes so that they are in alphabetical order. Each text child git's also normalized, by replacing all whitespace with a single space. Then run difflib over these, and count the number of diffrences. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Control webbrowser from Python script
Johny pyt...@hope.cz writes: On Oct 9, 5:17 pm, Tim Harig user...@ilthio.net wrote: On 2010-10-09, Johny pyt...@hope.cz wrote: Is it possible to control any webbrowser from Python ? For example to issue http POST and GET command The most reliable way to interact with a webserver is through the urllib and httplib modules. This is effective for 99% of cases. I do understand that some sites heavy in javascript can be difficult to figure out how to make valid requests. On win32 you can automate Internet Explorer through its InternetExplorer.Application COM interface which is fully documented at the MSDN site. On other operating systems, Firefox can be accessed using its XPCOM interface. Thanks ALL who replied.I wanted to use directly httplib modul and I started with it.But webserver uses SSL communication. Of course, I can use https protocol but the communication between server and client( browser) is so complicated that I was unable to simulate that with python script.So I think I could do that with webbrowser directly ( control webbrowser from a python script) What has HTTPS to do with this? My tip for this problem: use FireBug and HTTP Live Headers to get an idea what is really going on between Server Browser. Then model that with Python + urllib2. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: list parameter of a recursive function
TP tribulati...@paralleles.invalid writes: Diez B. Roggisch wrote: Back to your example: your solution is perfectly fine, although a bit costly and more error-prone if you happen to forget to create a copy. A safer alternative for these cases is using tuples, because they are immutable. Thanks Diez for your explanation. The problem with tuples is that it is not easy to modify them: in my case, I need to append a string to the tuple at each recursive step. So, it seems to me that I need to write a loop to modify the tuple, because on a simple example: a=(foo, bar) b=(a[0],a[1],toto) b (u'foo', u'bar', u'toto') I do not find any other means to obtain that result for b. With lists, I can use .extend(). Am I right? Yes and no - again. Of course you cannot use extend. But you can concatenate: b = a + (toto,) Note the trailing comma. A common misconception is to think that tuples are build using parethesis. They aren't. They are build using the comma-operator, and needed for disambiguation, e.g. in this case: foo(a, (b, c)) Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: how to write an xml file without dom.ext?
hackingKK hackin...@gmail.com writes: Hello all. I need to create an xml file. I am using dom.minidom module. It works fine as long as the xml tree is created. But I get the import error for dom.ext. I searched through the python docs but can't find a solution. I am pritty sure that there is way to write the file to disk without using the ext module. Since there are so many software doing this with python 2.6 I am sure it works. So can some one tell me if there is a way to avoide ext and prittyprint and still write a file to the disk? By not using minidom at all. from xml.etree import ElementTree as et doc = et.fromstring(mydoc/) print et.tostring(doc) Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: ConFoo spam?
Chris Withers ch...@simplistix.co.uk writes: Hi All, Is it just me or does the mailing of just about every single python-based project mailing list with a 90% form email advertising a conference that only has one python track *and* clashes with PyCon feel just a bit like spam? I know it's enough to put me off even considering going to ConFoo, whatever it is... The post appeared on TurboGears lists as well - but I actually consider it on-topic. We get enough other conference calls here, sometimes even without an actual python-track, but instead catering to specific scientific communities that might or might not use python. I don't consider this spam. It is certainly less annoying than the umpteenth GIL discussion... Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: hashkey/digest for a complex object
kj no.em...@please.post writes: In m2fwwjazs2@web.de de...@web.de (Diez B. Roggisch) writes: kj no.em...@please.post writes: The short version of this question is: where can I find the algorithm used by the tuple class's __hash__ method? Surprisingly, in the source: http://google.com/codesearch/p?hl=de#-2BKs-LW4I0/trunk/python/src/Objects/tupleobject.cq=python%20tuplehashsa=Ncd=1ct=rc Thanks to you, and to all who pointed me to the place in the source where this is. How exactly did you search for this? Taking a hint from the url above, I went to Google Code Search and searched for python tuple hash (minus the quotes, of course), which produced a ton of irrelevant stuff (almost 80K hits). Searching for python tuple hash lang:c cut down the number of hits to ~8K, but still too much to wade through. Clearly I need a smarter search strategy (one that does not include foreknowledge of the name of the actual function in the C source, of course). I tried codesearch first. Not satisfied after 30 seconds with the results, I did the most straight forward thing - I downloaded and un-packed the python source... and took a look. From that I learned the tuplehash function name. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: hashkey/digest for a complex object
kj no.em...@please.post writes: In 87pqvmp611@web.de de...@web.de (Diez B. Roggisch) writes: I tried codesearch first. Not satisfied after 30 seconds with the results, I did the most straight forward thing - I downloaded and un-packed the python source... and took a look. From that I learned the tuplehash function name. You must be at least somewhat familiar with the Python source. Everytime I've peeked into it I just feel lost, but it's clearly something I need to master sooner or later... I can't wait for the next one of those occasional introductions to the Python internals at our local Python club. No, I'm not the tiniest bit. I just followed my instincts in looking into the Objects folder, because that's where I suspected the definition of objects to be And grep has been proven useful in these cases as well. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Eclipse/PyDev - BOM Lexical Error
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes: In message 87d3rorf2f@web.de, Diez B. Roggisch wrote: Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes: What exactly is the point of a BOM in a UTF-8-encoded file? It's a marker like the coding: utf-8 in python-files. It tells the software aware of it that the content is UTF-8. But if the software is aware of it, then why does it need to be told? Let me rephrase: windows editors such as notepad recognize the BOM, and then assume (hopefully rightfully so) that the rest of the file is text in utf-8 encoding. So it is similar to the coding-header in Python. Naming it BOM is obviously stupid, but that's the way it is called. It is in fact a Unicode BOM character, and I can understand why it’s called that. What I’m trying to understand is why you need to put one in a UTF-8- encoded file. I hope that's clear now. It says I'm a UTF-8 file. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: help!!!
fkr...@aboutrafi.net23.net writes: plz can u convert this cpp file into python i need that badly as soon as possible... I am new to python. I just wanna learn it For such an aspiring student of the art of computer programming, I have the strange feeling of lack-of-effort-showing here. Do I have to lose my faith in youth? Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: suggestions please what should i watch for/guard against' in a file upload situation?
Seebs usenet-nos...@seebs.net writes: On 2010-10-06, geekbuntu gmi...@gmail.com wrote: in general, what are things i would want to 'watch for/guard against' in a file upload situation? This question has virtually nothing to do with Python, which means you may not get very good answers. In contrast to comp.super.web.experts? There are quite a few people with web-experience here I'd say. my checklist so far is basically to check the extension - ensure it has 3 places, ensure it's in the allowed list (like jpg gif etc...). This strikes me as 100% irrelevant. Who cares what the extension is? Given that most people are not computer savvy (always remember, the default for windows is to hide extensions..), using it client-side can be valuable to prevent long uploads that eventuall need to be rejected otherwise (no mom, you can't upload word-docs as profile pictures). not sure what else i could do to guard against anything bad happening. maybe the file name itself could cause greif? Obvious things: * File name causes files to get created outside some particular upload directory (../foo) Or rather just store that as a simple meta-info, as allowing even the best-intended me-in-cool-pose.jpg to overwrite that of the one other cool guy using the website isn't gonna fly anyway. * File name has spaces See above, but other then that - everything but shell-scripts deal well with it. * Crazy stuff like null bytes in file name * File names which might break things if a user carelessly interacts with them, such as foo.jpg /etc/passwd bar.jpg (all one file name including two spaces). Your strange focus on file-names that are pure meta information is a little bit concerning... Basically, the key question is, could a hostile user come up with input to your script which could break something? Certainly advice. But that's less focussed on filenames or file-uploads, but on the whole subject of processing HTTP-requestst. Which would make a point for *not* using a home-grown framework. But then, Python is a bit less likely to suffer from buffer overflow or similar kind of attacks. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: hashkey/digest for a complex object
kj no.em...@please.post writes:
The short version of this question is: where can I find the algorithm
used by the tuple class's __hash__ method?
Surprisingly, in the source:
http://google.com/codesearch/p?hl=de#-2BKs-LW4I0/trunk/python/src/Objects/tupleobject.cq=python%20tuplehashsa=Ncd=1ct=rc
Now, for the long version of this question, I'm working with some
complext Python objects that I want to be able to compare for
equality easily.
These objects are non-mutable once they are created, so I would
like to use a two-step comparison for equality, based on the
assumption that I can compute (either at creation time, or as needed
and memoized) a hashkey/digest for each object. The test for
equality of two of these objects would first compare their hashkeys.
If they are different, the two objects are declared different; if
they match, then a more stringent test for equality is performed.
So the problem is to come up with a reasonable hashkey for each of
these objects. The objects have two significant attributes, and
two of these objects should be regarded as equal if these attributes
are the same in both. The first attribute is a simple dictionary
whose keys are integers and values are strings. The second attribute
is more complicated. It is a tree structure, represented as a
dictionary of dictionaries of dictionaries... until we get to the
leaf elements, which are frozensets of strings. The keys at every
level of this structure are strings. E.g. a simple example of such
an attribute would look like:
{'A': {'a': set(['1', '2', '3']),
'b': set(['4', '5'])},
'B': set(['6', '7', '8'])}
I'm looking for a good algorithm for computing a hash key for
something like this? (Basically I'm looking for a good way to
combine hashkeys.)
Creating tuples from dicts, recursively, and stabilized by using sorted
on items.
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: list parameter of a recursive function
TP tribulati...@paralleles.invalid writes: Hi, I have a function f that calls itself recursively. It has a list as second argument, with default argument equal to None (and not [], as indicated at: http://www.ferg.org/projects/python_gotchas.html#contents_item_6 ) This is the outline of my function: def f ( argument, some_list = None ): if some_list == None: some_list = [] [...] # creation of a new_argument # the function is called recursively only if some condition is respected if some_condition: some_list.append( elem ) f( new_argument, some_list ) # otherwise, we have reached a leaf of the a branch of the recursive tree # (said differently, terminal condition has been reached for this branch) print Terminal condition The problem is that when the terminal condition is reached, we return back to some other branch of the recursive tree, but some_list has the value obtained in the previous branch! So, it seems that there is only one some_list list for all the recursive tree. To get rid of this behavior, I have been compelled to do at the beginning of f: import copy from copy some_list = copy( some_list ) I suppose this is not a surprise to you: I am compelled to create a new some_list with the same content. So, if I am right, all is happening as if the parameters of a function are always passed by address to the function. Whereas in C, they are always passed by copy (which gives relevance to pointers). Am I right? Yes and no. For some sensible definition of yes (this is to avoid the *endless* discussions about python parameter passing semantics and it's relation to common, yet misunderstood or unprecisely defined definitions of parameter passing), passing around a mutable object (such as a list or dictionary) will be like a pointer - mutations are visible to all others having a reference to it. You are wrong though that C-semantics are different, meaning that passing by copy is not what happens. Some things are copied (primitive types, structs that are passed directly). But not, for example - and relevant to this case - arrays (as they are only pointers), lists (as they don't exist, but are modeled by structs containing pointers). Back to your example: your solution is perfectly fine, although a bit costly and more error-prone if you happen to forget to create a copy. A safer alternative for these cases is using tuples, because they are immutable. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: suggestions please what should i watch for/guard against' in a file upload situation?
Seebs usenet-nos...@seebs.net writes: On 2010-10-06, Diez B. Roggisch de...@web.de wrote: Seebs usenet-nos...@seebs.net writes: On 2010-10-06, geekbuntu gmi...@gmail.com wrote: in general, what are things i would want to 'watch for/guard against' in a file upload situation? This question has virtually nothing to do with Python, which means you may not get very good answers. In contrast to comp.super.web.experts? There are quite a few people with web-experience here I'd say. Oh, certainly. But in general, I try to ask questions in a group focused on their domain, rather than merely a group likely to contain people who would for other reasons have the relevant experience. I'm sure that a great number of Python programmers have experience with sex, that doesn't make this a great newsgroup for sex tips. (Well, maybe it does.) As the OP asked about a Python web framework (self written or not), I think all advice that can be given is certainly more related to Python than to airy references to general web programming such as oh, make sure if your server side application environment hasn't any security issues. Or, to be more concrete: what NG would you suggest for frameworks or webapps written in python to ask this question? Given that most people are not computer savvy (always remember, the default for windows is to hide extensions..), using it client-side can be valuable to prevent long uploads that eventuall need to be rejected otherwise (no mom, you can't upload word-docs as profile pictures). That's a good point. On the other hand, there's a corollary; you may want to look at the contents of the file in case they're not really what they're supposed to be. For sure. But the focus of you and others seems to be the file-name, as if that was anything especially dangerous. Matter of factly, it's a paramteter to a multipart/form-data encoded request body parameter definition, and as such has a rather locked-down in terms of null-bytes and such. So you are pretty safe as long as you - use standard library request parsing modules such as cgi. If one instist on reading streams bytewise and using ctypes to poke the results into memory, you can of course provoke unimaginable havoc.. - don't use the filename for anything but meta-info. And ususally, they are simply regarded as nice that you've provided us with it, we try make our best to fill an img alt attribute with the basename. But not more. Worth pointing out to the OP to do that. But this is *not* a matter of mapping HTTP-request paths to directories I'd wager to say. Something that is of much more importance (I should have mentioned earlier, shame on me) is of course file-size. Denying requests that come with CONTENT_LENGTH over a specified limit, of course respecting CONTENT_LENGTH and not reading beyond it, and possibly dealing with chunked-encodings in similarily safe ways (I have to admit I haven't yet dealt with one of those myself on a visceral level - but as they are part of the HTTP-spec...) is important, as otherwise DOS attacks are possible. Your strange focus on file-names that are pure meta information is a little bit concerning... If you're uploading files into a directory, then it is quite likely that you're getting file names from somewhere. Untrusted file names are a much more effective attack vector, in most cases, than EXIF information. The into a directory quote coming from where? And given that EXIF information is probably read by some C-lib, I'd say it is much more dangerous. This is a gut feeling only, but fed by problems with libpng a year or two ago. Certainly advice. But that's less focussed on filenames or file-uploads, but on the whole subject of processing HTTP-requestst. Which would make a point for *not* using a home-grown framework. Well, yeah. I was assuming that the home-grown framework was mandatory for some reason. Possibly a very important reason, such as otherwise we won't have written it ourselves. In Python, it's usually more along the lines of well, we kinda started, and now we have it, and are reluctant to switch. But of course one never knows... Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: suggestions please what should i watch for/guard against' in a file upload situation?
Martin Gregorie mar...@address-in-sig.invalid writes: On Wed, 06 Oct 2010 09:02:21 -0700, geekbuntu wrote: in general, what are things i would want to 'watch for/guard against' in a file upload situation? i have my file upload working (in the self-made framework @ work without any concession for multipart form uploads), but was told to make sure it's cleansed and cannot do any harm inside the system. Off the top of my head, and assuming that you get passed the exact filename that the user entered: - The user may need to use an absolute pathname to upload a file that isn't in his current directory, so retain only the basename by discarding the rightmost slash and everything to the left of it: /home/auser/photos/my_photo.jpg === my_photo.jpg c:\My Photos\My Photo.jpg === My Photo.jpg - If your target system doesn't like spaces in names or you want to be on the safe side there, replace spaces in the name with underscores: My Photo.jpg ===My_Photo.jpg - reject any filenames that could cause the receiving system to do dangerous things, e.g. .EXE or .SCR if the upload target is Windows. This list will be different for each upload target, so make it configurable. Erm, this assumes that the files are executed in some way. Why should they? It's perfectly fine to upload *anything*, and of course filenames mean nothing wrt to the actual file contents (Are you sure you want to change the extension of this file?). It might make no sense for the user, because you can't shon an exe as profile image. But safe-guarding against that has nothing to do with OS. And even safe file formats such as PNGs have been attack vectors. Precisely because they are processed client-side in the browser through some library with security issues. For serving the files, one could rely on the file-command or similar means to determine the mime-type. So far, I've never done that - as faking the extension for something else doesn't buy you something unless there is a documented case of internet explorer ignoring mime-type, and executing downloaded file as program. You can't assume anything about else about the extension. .py .c .txt and .html are all valid in the operating systems I use and so are their capitalised equivalents. - check whether the file already exists. You need rules about what to do if it exists (do you reject the upload, silently overwrite, or alter the name, e.g. by adding a numeric suffix to make the name unique: my_photo.jpg === my_photo-01.jpg Better, associate the file with the uploader and or it's hash. Use the name as pure meta-information only. There's probably something I've forgotten, but that list should get you going. Dealing with to large upload requests I'd say is much more important, as careless reading of streams into memory has at least the potential for a DOS-attack. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: help!!!
Seebs usenet-nos...@seebs.net writes: On 2010-10-06, Diez B. Roggisch de...@web.de wrote: fkr...@aboutrafi.net23.net writes: plz can u convert this cpp file into python i need that badly as soon as possible... I am new to python. I just wanna learn it For such an aspiring student of the art of computer programming, I have the strange feeling of lack-of-effort-showing here. Do I have to lose my faith in youth? Never! Just be sure you are having faith in them to, well, be youth. :) I took the freedom to google a bit. If one can trust people on the inter-tubes (and who would have heard you can't?), he's a rather skilled student http://www.aboutrafi.net23.net/ With an impressive amount of technological experience under his belt. So I'm a bit aghast to see him struggle with this rather simple problem. Thus my question... Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: help!!!
Seebs usenet-nos...@seebs.net writes: On 2010-10-06, Diez B. Roggisch de...@web.de wrote: With an impressive amount of technological experience under his belt. So I'm a bit aghast to see him struggle with this rather simple problem. Thus my question... It does seem a bit odd. I mean, if I had a short time line to get something converted to Python, I'd probably ask for help, but then, I wouldn't take a gig where I needed to quickly move something into an unfamiliar language. I'm happily puttering away slowly at converting a couple of things to Python to get the hang of it, carefully selecting things with no impending deadlines. From the look of it... he's just trying to get a freebie on a homework assignment. It certainly is no commercial/useful piece of code whatsoever... just CS-101 read data from stdin and do stuff. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Eclipse/PyDev - BOM Lexical Error
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes: In message a6080839-c74d-4466-aa21-f8c4827cb...@y12g2000prb.googlegroups.com, TheOne wrote: I want the source files to have BOM character. What exactly is the point of a BOM in a UTF-8-encoded file? It's a MS-specific thing that makes a file identifieable as UTF-8-encoded under windows. The name BOM is obviously BS, but it's the way it is. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Eclipse/PyDev - BOM Lexical Error
TheOne daewon.y...@gmail.com writes: On Oct 4, 9:26 pm, de...@web.de (Diez B. Roggisch) wrote: TheOne daewon.y...@gmail.com writes: Hi. I installed eclipse/pydev today. I created a pydev project and added python source files with utf-8 BOM. Eclipse/Pydev reports lexical error : Lexical error at line 1, column 1. Encountered: \ufeff (65279), after : I want the source files to have BOM character. How could I shut off this error msg? No idea. Why do you want it? Is somebody else processing these scripts? If it's about declaring them to be utf-8, you should consider placing # -*- coding: utf-8 -*- on the first or second line. That works for python, and should work for eclipse. Diez I also included that # -*- coding: line. I just don't want me or other project members to accidentally save them in different encoding. So I (and my team) thought it would be safe to have the BOM character. Well, me and my team we don't have it, but rarely if ever (can't remember when) issues with this. And we are using vim, emacs + pydev. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem installing psycopg2 in virtualenv (Ubuntu 10.04, Python 2.5)
Pascal Polleunus p...@especific.be writes: On 05/10/10 00:11, Diez B. Roggisch wrote: Pascal Polleunusp...@especific.be writes: Hi, I've problems to install psycopg2 in a virtualenv on Ubuntu 10.04. My problem is also explained on stackoverflow: http://stackoverflow.com/questions/3847536/installing-psycopg2-in-virtualenv-ubuntu-10-04-python-2-5 I tried different things explained there: http://www.saltycrane.com/blog/2009/07/using-psycopg2-virtualenv-ubuntu-jaunty/ The last thing I tried is this... I created a virtualenv with -p python2.5 --no-site-packages I installed libpq-dev: apt-get install libpq-dev In the virtualenv, I did this: easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base Then when I tried pip install psycopg2==2.0.7, I got this error: Installing collected packages: psycopg2 Running setup.py install for psycopg2 building 'psycopg2._psycopg' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION=2.2.2 (dt dec ext pq3) -DPG_VERSION_HEX=0x080404 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -DHAVE_PQPROTOCOL3=1 -I/usr/include/python2.5 -I. -I/usr/include/postgresql -I/usr/include/postgresql/8.4/server -c psycopg/psycopgmodule.c -o build/temp.linux-i686-2.5/psycopg/psycopgmodule.o -Wdeclaration-after-statement psycopg/psycopgmodule.c:27:20: error: Python.h: No such file or directory In file included from psycopg/psycopgmodule.c:31: ./psycopg/python.h:31:26: error: structmember.h: No such file or directory ./psycopg/python.h:34:4: error: #error psycopg requires Python= 2.4 In file included from psycopg/psycopgmodule.c:32: Does anyone have any idea how to solve that? Install the python-dev-package. It contains the Python.h file, which the above error message pretty clearly says. Usually, it's a good idea to search package descriptions of debian/ubuntu packages for missing header files to know what to install. It's already installed; at least for 2.6, nor sure it's correct for 2.5. python2.5-dev is not available but python-old-doctools replaces it. It is 100% *not* correct for python2.5. As the error message shows - it's missing. If it's not available somewhere, you should consider building python yourself, if you have to use 2.5. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Eclipse/PyDev - BOM Lexical Error
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes: In message 2752e2e4-76fe-475a-a476-e5458bbfd...@z30g2000prg.googlegroups.com, TheOne wrote: Anyway, it would be great if I could make my eclipse/pydev to understand the BOM character and suppress the lexical error msg. What exactly is the point of a BOM in a UTF-8-encoded file? It's a marker like the coding: utf-8 in python-files. It tells the software aware of it that the content is UTF-8. Naming it BOM is obviously stupid, but that's the way it is called. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: direct print to log file
Dirk Nachbar dirk...@gmail.com writes: How can I direct all print to a log file, eg some functions have their own print and I cannot put a f.write() in front of it. you can replace sys.stdout with something that performs logging. class MyWriter(object): def __init__(self, old_stream): self.old_stream = old_stream self.logger = logging.getLogger(stdout) def write(self, msg): self.old_stream.write(msg) self.logger.debug(msg) sys.stdout = MyWriter(sys.stdout) Untested - but you get the gist. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: subclass constructor problem
Nikola Skoric n...@fly.srk.fer.hr writes: I have a superclass Element and a subclass Family. All Family.__init__() does is run Element.__init__() and self.__parse(). For some reason it seems like self.__parse() isn't run. Here is the code: http://github.com/dijxtra/simplepyged/blob/8d5a6d59268f6760352783cce6b97c0b5e75b8be/simplepyged/simplepyged.py In this version everything works fine. Notice lines 698, 703 and 708. If I remove those lines, those 3 methods return None or []. Seems like __init__ won't run __parse, but other methods run it without problem. So, what obvious thing am I missing? :-) Works without a hitch for my condensed example: class Element(object): def __init__(self): pass class Family(Element): def __init__(self): Element.__init__(self) self.__parse() def __parse(self): self.parse = Family.__parse f = Family() print f.parse Btw, you are a bit on the overprotective side. The convention for marking attributes (methods or objects alike) private is by prefixing them with a *single* underscore. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: ElementTree handling nested tag
tekion tek...@gmail.com writes: On Oct 3, 2:09 pm, de...@web.de (Diez B. Roggisch) wrote: tekion tek...@gmail.com writes: On Oct 2, 5:32 am, de...@web.de (Diez B. Roggisch) wrote: tekion tek...@gmail.com writes: All, I have the following xml tag: event resource_access actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse /resource_access /event I am interested in: actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse as well as the upper layer tag. How do I get at the nest tag listed above? Thanks. What is the upper layer tag? And what do you actually want to get? The text-values? Or do you want to just create a document that just consists of the resource_access tag? Then this should help: from xml.etree.ElementTree import * doc = event resource_access actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse /resource_access /event doc = fromstring(doc) resource_access = doc.find(resource_access) print tostring(resource_access) Diez Diez, This is the sample format from the doc. I the whole log file has this xml formated beginning and ending in the event tag. Is this something ElemenTtree can handle or is it better to user XSLT? Thanks. Handle *what*? Can it read it? Yes. Can it extract data from it? Yes. You still haven't said what you actually *want* with all this. Diez I wan to get the value of these tags: httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse from xml.etree.ElementTree import * doc = event resource_access actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse /resource_access /event doc = fromstring(doc) for resource_access in doc.findall(resource_access): print resource_access.find(httpurl).text You asked for what the upper layer tags are. The upper layer tags I am referring for below tags are any tag that is above it. httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse IE, for the upper upper are: actionhttpRequest/action and event tags. That's not true. action is on the same level as the others. And what you you want to get from the upper layer tags? Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a Python Version Manager?
TerryP bigboss1...@gmail.com writes: On Oct 4, 4:12 am, Kushal Kumaran kushal.kumaran+pyt...@gmail.com wrote: Is virtualenv what you need? http://pypi.python.org/pypi/virtualenv snip -- regards, kushal Not quite. It basically amounts to a UNIX version of xcopy'ing an existing Python installation. ... install Python X.Y by any means python virtualenv.py -p=X.Y whereToStoreFiles # clone existing Python X.Y install python virtualenv.py --relocatable whereToStoreFiles # make xcopy/ tar friendly . whereToStoreFiles/bin/activate python file.py args # run using whereToStoreFiles/bin/python cloned by above ... rinse and repeat for each X.Y versus . ./path/to/rvm/script rvm install 1.8.7,1.9.2,rbx # fetch and install Ruby 1.8.7, 1.9.2, and Rubinius in rvm root rvm 1.9.2 some-ruby-command args # run using Ruby 1.9.2. python2.5 rvm 1.8.7 some-other-rbcmd args # run using Ruby 1.8.7 python2.4 rvm --default rbx # set default ruby for this shell alias python=python2.5 ruby file.rb args # use rubininus as ruby ruby system # use systems ruby instead /usr/bin/python ruby file.rb args # ^ Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Eclipse/PyDev - BOM Lexical Error
TheOne daewon.y...@gmail.com writes: Hi. I installed eclipse/pydev today. I created a pydev project and added python source files with utf-8 BOM. Eclipse/Pydev reports lexical error : Lexical error at line 1, column 1. Encountered: \ufeff (65279), after : I want the source files to have BOM character. How could I shut off this error msg? No idea. Why do you want it? Is somebody else processing these scripts? If it's about declaring them to be utf-8, you should consider placing # -*- coding: utf-8 -*- on the first or second line. That works for python, and should work for eclipse. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem installing psycopg2 in virtualenv (Ubuntu 10.04, Python 2.5)
Pascal Polleunus p...@especific.be writes: Hi, I've problems to install psycopg2 in a virtualenv on Ubuntu 10.04. My problem is also explained on stackoverflow: http://stackoverflow.com/questions/3847536/installing-psycopg2-in-virtualenv-ubuntu-10-04-python-2-5 I tried different things explained there: http://www.saltycrane.com/blog/2009/07/using-psycopg2-virtualenv-ubuntu-jaunty/ The last thing I tried is this... I created a virtualenv with -p python2.5 --no-site-packages I installed libpq-dev: apt-get install libpq-dev In the virtualenv, I did this: easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base Then when I tried pip install psycopg2==2.0.7, I got this error: Installing collected packages: psycopg2 Running setup.py install for psycopg2 building 'psycopg2._psycopg' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION=2.2.2 (dt dec ext pq3) -DPG_VERSION_HEX=0x080404 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -DHAVE_PQPROTOCOL3=1 -I/usr/include/python2.5 -I. -I/usr/include/postgresql -I/usr/include/postgresql/8.4/server -c psycopg/psycopgmodule.c -o build/temp.linux-i686-2.5/psycopg/psycopgmodule.o -Wdeclaration-after-statement psycopg/psycopgmodule.c:27:20: error: Python.h: No such file or directory In file included from psycopg/psycopgmodule.c:31: ./psycopg/python.h:31:26: error: structmember.h: No such file or directory ./psycopg/python.h:34:4: error: #error psycopg requires Python = 2.4 In file included from psycopg/psycopgmodule.c:32: Does anyone have any idea how to solve that? Install the python-dev-package. It contains the Python.h file, which the above error message pretty clearly says. Usually, it's a good idea to search package descriptions of debian/ubuntu packages for missing header files to know what to install. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Deditor:Pythonic text editor
Kruptein darragh@gmail.com writes: Hey, I released the 0.2.1 version of my text-editor written for linux in python using the wxPython toolkit. I would like to know whether it is good/bad and what could be changed/ added this version added -syntax-highlighting for 78 languages -Tab completion in the filepath bar -Shortcut customizer: you can choose the keybindings yourself you can download a deb or tar.gz file from http://launchpad.net/deditor Some screenshots would be nice. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: ElementTree handling nested tag
tekion tek...@gmail.com writes: On Oct 2, 5:32 am, de...@web.de (Diez B. Roggisch) wrote: tekion tek...@gmail.com writes: All, I have the following xml tag: event resource_access actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse /resource_access /event I am interested in: actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse as well as the upper layer tag. How do I get at the nest tag listed above? Thanks. What is the upper layer tag? And what do you actually want to get? The text-values? Or do you want to just create a document that just consists of the resource_access tag? Then this should help: from xml.etree.ElementTree import * doc = event resource_access actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse /resource_access /event doc = fromstring(doc) resource_access = doc.find(resource_access) print tostring(resource_access) Diez Diez, This is the sample format from the doc. I the whole log file has this xml formated beginning and ending in the event tag. Is this something ElemenTtree can handle or is it better to user XSLT? Thanks. Handle *what*? Can it read it? Yes. Can it extract data from it? Yes. You still haven't said what you actually *want* with all this. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: ElementTree handling nested tag
tekion tek...@gmail.com writes: All, I have the following xml tag: event resource_access actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse /resource_access /event I am interested in: actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse as well as the upper layer tag. How do I get at the nest tag listed above? Thanks. What is the upper layer tag? And what do you actually want to get? The text-values? Or do you want to just create a document that just consists of the resource_access tag? Then this should help: from xml.etree.ElementTree import * doc = event resource_access actionhttpRequest/action httpurlHTTP://cmd.wma.ibm.com:80//httpurl httpmethodGET/httpmethod httpresponse200/httpresponse /resource_access /event doc = fromstring(doc) resource_access = doc.find(resource_access) print tostring(resource_access) Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem saving uploaded files in Python3
Chris Rebert c...@rebertia.com writes:
On Fri, Oct 1, 2010 at 11:13 PM, hid...@gmail.com wrote:
Hello, i control the problem of the data what is uploaded by the POST
method, in the web if the file is a text theres no problem
but the trouble comes when it's an enconded file as a Picture or other what
the when the system insert the data into the file
well it doesn 't encoded in the write way i will put all the code, from the
area whats take the environ['wsgi.input'] to the area
thats save the file:
tmpData = str(rawData)[1:].strip(' ')#Here the data from the
environ['wsgi.input'], first i convert the byte into a string delete the
first field that represent the b and after i strip the single quotes
dat = tmpData.split('\\r')#Then i split all the data in the '\\r'
s = open('/home/hidura/test.png', 'w')#I open the test.png file.
snip
Where is the mistake?
In believing that wsgi.input is a simple file that contains your
upload. It isn't. It is multipart/form-data encoded, to cope with
potentially many files being uploaded in one go, plus assorted other
form fields.
In another post, you have already been shown how to do this. What where
the results? How does your full script look like? How does the form look
like you upload with?
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: Unclear datetime.date type when using isinstance
Mailing List li...@fastmail.net writes: Was including a input check on a function argument which is expecting a datetime.date. When running unittest no exception was raised when a datetime.datetime instance was used as argument. Some playing with the console lead to this: import datetime dt1 = datetime.datetime(2010, 10, 2) type(dt1) type 'datetime.datetime' isinstance(dt1, datetime.datetime) True isinstance(dt1, datetime.date) True dt2 = datetime.date(2010, 10, 2) type(dt2) type 'datetime.date' isinstance(dt2, datetime.datetime) False isinstance(dt2, datetime.date) True My issue (or misunderstanding) is in the first part, while dt1 is a datetime.date object (confirmed by type), the isinstance(dt1, datetime.datetime) returns True. Is this correct? If so, is it possible to verify whether an object is indeed a datetime.date and not a datetime.datetime object? I think you got it wrong here: dt1 is a *datetime* object, and also a date-object. I guess that's because it conforms to the same protocol - it has year, month and day and so forth. For background information; reason my choice was to use the datetime.date object is that there should not be ay time information in the object. Of course this can easily be stripped off in the function, but thought the use of a datetime.date would be a nice shortcut... Obviously not :) You can of course test not on inheritance (which isinstance does), but actual type equality: type(dt1) == date False Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: utf-8 and ctypes
Brendan Miller catph...@catphive.net writes: 2010/9/29 Lawrence D'Oliveiro l...@geek-central.gen.new_zealand: In message mailman.1132.1285714474.29448.python-l...@python.org, Brendan Miller wrote: It seems that characters not in the ascii subset of UTF-8 are discarded by c_char_p during the conversion ... Not a chance. ... or at least they don't print out when I go to print the string. So it seems there’s a problem on the printing side. What happens when you construct a UTF-8-encoded string directly in Python and try printing it the same way? Doing this seems to confirm something is broken in ctypes w.r.t. UTF-8... if I enter: str = 日本語のテスト What is this? Which encoding is used by your editor to produce this byte-string? If you want to be sure you have the right encoding, you need to do this: - put a coding: utf-8 (or actually whatever your editor uses) in the first or second line - use unicode literals. That are the funny little strings with a u in front of them. They will be *decoded* using the declared encoding. - when passing this to C, explicitly *encode* with utf-8 first. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: will Gnome 3.0 kill pygtk?
Tracubik affdfsdfds...@b.com writes: Hi! It seem that the new version of gnome 3.0 will dismiss pygtk support. link: [1] http://live.gnome.org/TwoPointNinetyone/ (search killing pygtk) [2] http://live.gnome.org/GnomeGoals/PythonIntrospectionPorting i'm studying pygtk right now, am i wasting my time considering that my preferred platform is linux/gnome? The important bit is that it will not ditch Python-support. I can't comment at the actual differences between pygtk and the upcoming reflection-based approach, but I doubt learning pygtk now is a waste. First of all, it will be quite a while until pygtk (and gnome 2.x) are really gone. And even if, the concepts stay the same, so unless pygtk adds an extra layer of abstraction (which AFAIK it doesn't), your knownledge is still usable - if not translatable one-to-one. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Problems with wsgi Python3
hid...@gmail.com writes: Hello list, i had seriously troubles with the connection between a form and the wsgi, i' ve made an application on Python3 and was running perfectly but when i try to use the form to pass the data this can't be see on the server, so what is your recommendation?, i am open to all the ideas less leave to Python2 i can't all the project was write on Python3, i was looking into the PEP 444 proposal too. Here is the code what i used to take the data: codeprint(urllib.parse.parse_qs(environ['wsgi.input'].read(int(environ ['CONTENT_LENGTH'])).decode()))/code What is the ouptut of this? What does import pprint pprint.pprint(environ) give you? and here it's what the os.environ.item() prints to me: That's irrelevant. The WSGI-environ is what is needed. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Problems with wsgi Python3
] 'mod_wsgi.script_reloading': [Thu Sep 30 13:35:07 2010] [error] '1', [Thu Sep 30 13:35:07 2010] [error] 'mod_wsgi.version': [Thu Sep 30 13:35:07 2010] [error] (3, 3), [Thu Sep 30 13:35:07 2010] [error] 'wsgi.errors': [Thu Sep 30 13:35:07 2010] [error] , [Thu Sep 30 13:35:07 2010] [error] 'wsgi.file_wrapper': [Thu Sep 30 13:35:07 2010] [error] , [Thu Sep 30 13:35:07 2010] [error] 'wsgi.input': [Thu Sep 30 13:35:07 2010] [error] , [Thu Sep 30 13:35:07 2010] [error] 'wsgi.multiprocess': [Thu Sep 30 13:35:07 2010] [error] False, [Thu Sep 30 13:35:07 2010] [error] 'wsgi.multithread': [Thu Sep 30 13:35:07 2010] [error] True, [Thu Sep 30 13:35:07 2010] [error] 'wsgi.run_once': [Thu Sep 30 13:35:07 2010] [error] False, [Thu Sep 30 13:35:07 2010] [error] 'wsgi.url_scheme': [Thu Sep 30 13:35:07 2010] [error] 'http', [Thu Sep 30 13:35:07 2010] [error] 'wsgi.version': [Thu Sep 30 13:35:07 2010] [error] (1, 1)} On Sep 30, 2010 1:56pm, Diez B. Roggisch de...@web.de wrote: *sigh* AGAIN, please do *not* reply to me personally. Reply to the ML or NG you are posting in. I'm beginning to believe that it's a waste of time to answer you since you seem not able to follow even the most simple of advice and rules. And read the answers you get thoroughly, you did not give all the information I asked you for. On Sep 30, 2010, at 7:08 PM, hid...@gmail.com wrote: The output is: -110550484811701713451664404475--\r No more i don 't know what that means or what do with this. On Sep 30, 2010 11:55am, Diez B. Roggisch de...@web.de wrote: hid...@gmail.com writes: Hello list, i had seriously troubles with the connection between a form and the wsgi, i' ve made an application on Python3 and was running perfectly but when i try to use the to pass the data this can't be see on the server, so what is your recommendation?, i am open to all the ideas less leave to Python2 i can't all the project was write on Python3, i was looking into the PEP 444 proposal too. Here is the code what i used to take the data: print(urllib.parse.parse_qs(environ['wsgi.input'].read(int(enviro n ['CONTENT_LENGTH'])).decode())) What is the ouptut of this? What does import pprint pprint.pprint(environ) give you? and here it's what the os.environ.item() prints to me: That's irrelevant. The WSGI-environ is what is needed. Diez -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Example or recomendation of a webserver
Hidura hid...@gmail.com writes: I am working on a web project written on Py3k and using mod_wsgi on the Apache that have to recibes the request client via a xml structure and i am facing a lot of troubles with the upload files mainly because i can' t see where they are, so i' ve decide to write my own web server-django and the others doen' t work for my propouse-, what you recomend me for start that? not doing it. Reading the HTTP-RFC and the WSGI-PEP. And trying print environ[wsgi.input].read() inside your wsgi-script. Then, use webob to wrap the whole WSGI-stuff to have at least a minimum of sensible abstraction. Or simply use Django or TurboGears2, follow the advices in their docs on how to depoly them using mod_wsgi, and be happy. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: relative imports and sub-module execution
King animator...@gmail.com writes: Hi, After reading couple of docs and articles, I have implemented a simple test package with nested modules. When running main.py, everything is working fine. Some of my sub- modules has some small test routines for debug purpose. It's because I am using relative package imports at the top, I am not able to run these sub modules individually. For example, this works when running from main.py Here is example structure main.py __init__.py core/ __init__.py folder1 __init__.py f1a.py f1b.py folder2 __init__.py f2a.py f2b.py in folder2/f2b.py, I am importing from core.folder1 import f1a print f1a.myvar Now I can't execute 'f2b.py' individually. It's giving an error: ImportError: No module named core.folder2 Is this mean when you have created a package where modules are using relative imports, they can't execute individually? The problem is your python-path. Python will put the path of the script that you execute into the sys.path - but not attempt to guess that you actually are deep within a package hierarchy and want the path two up from there in sys.path as well. Which is the reason why you have to do that yourself. My solution to this is to always use setuptools, even for the tiniest of projects, and simply create a setup.py with some minimal information in it, and then do python setup.py develop This will put a EGG-link into the site-packages directory. Usually, for not clobbering my system's python, I use a virtualenv also. With this setup, I can safely import from core all the time. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: function decorators
Nick Donohue ndono...@gmail.com writes: I came across this code just now: def time_me(function): def wrap(*arg): start = time.time() r = function(*arg) end = time.time() print %s (%0.3f ms) %(function.func_name, (end-start)*1000) return wrap @time_me def some_function(somearg) some_function(arg) I've been looking online about what I think is going on, and from what I can tell this code is using function decorators. I guess what I'm asking is if someone could tell me what exactly is going on in this code - how is it different from passing: time_me(some_function(123))? I've tried it this way and it works. This is *not* what the decorator is doing. The equivalent of a decorator and then calling the result is this: some_function = time_me(some_function) some_function(123) Notice the difference? The decorator (a badly written one, by the way - it doesn't deal with a possible return value and keyword args) wraps the function into time measuring code. In general, a decorator is a callable that takes one argument. And whatever that callable returns is then bound under the same name as the original function (or class, since python 2.6 I believe) Which is what some_function = time_me(some_function) actually does. So decorator syntax with the @ is really just that - a bit of syntactic sugar. why would I use these? wouldn't it be more flexible to not write the decorator before the function definition, so I could choose to wrap it or not? Of course it's more flexible to do whatever the decorator does only if you need it. If you need that, it's a sign of a misuse of decorators. Their strength is in making boiler-plate code run without you having to type it out all the time. E.g. putting transactional boundaries around some code that deals with a database. Checking for proper authentication and authorization. Uniformely deal with exceptions. And so forth. Amongst the more popular decorators in python are the classmethod and property decorators. Go look them up in the stdlib. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Check whether file is being written to
Thomas Jollans tho...@jollybox.de writes: On Thursday 23 September 2010, it occurred to loial to exclaim: How can I check whether a file is being written to by another process before I access it? Platform is unix. As such, you can't. But you can lock the file using the functions in the fcntl module. Last time I checked, file-locking in unix was co-operative. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: creating python daemon ?
vineet daniel vineetdan...@gmail.com writes:
On Sep 21, 9:47 pm, de...@web.de (Diez B. Roggisch) wrote:
vineet daniel vineetdan...@gmail.com writes:
Hi
I have succesfully created daemon with python script and as next step
I am trying to give input to that python script daemon from Apache
Logshere I have got stuck and I have even checked IRC python
channel for solution. Apache is able to call the file but fails to
execute it properly and I get this error continuosly :
piped log program '/var/www/html/test.py' failed unexpectedly
How do I rectify the above error and make adjustment to the code so
that it takes input from apache.
code that I am using is as follows :
#! /usr/bin/env python
import sys,os
pid = os.fork()
os.chdir(/var/www/html/)
os.setsid()
os.umask(0)
#this line will read apache log in real time as redirected from the
CustomLog directive of Apache.
log = sys.stdin.readlines()
f = open('/var/www/logmongo.txt','a') # append log to this text file
f.write(log)
I'd appreciate if anybody could share the code that they used for
daemon or used with Apache CustomLog directive.
The above code looks errornous - you don't check for the return-value of
PID take appropriate action. Like
There is a daemonization recipe on active-state, which works nicely for
me.
http://code.activestate.com/recipes/278731-creating-a-daemon-the-pyth...
Diez
Hi Diez
Thanks for pointing that out.
Ever tried giving input to such python daemons from a dynamic source
like Apache logs which get generated in real time. I want apache to
directly write to this python daemon which in turn will process the
logs the way I want. Any help will help me immensely.
I don't understand the question. How is that input produced? How shall
the two processes interact? Through some logfile one writes, the other
reads? Through RPC-mechanisms? Sockets? Pipes?
The fact that process B is a daemon or not has nothing to do with this
whatsoever, btw.
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python in Linux - barrier to Python 3.x
Ant ant...@gmail.com writes: Hi all, I've just seen this: http://sheddingbikes.com/posts/1285063820.html Whatever you think of Zed Shaw (author of the Mongrel Ruby server and relatively recent Python convert), he has a very good point in this. I run Fedora 12 on my home computers, and find it far too much hassle to try to get Python 3 installed. Even the 2.x's are behind - IIRC think it currently uses 2.5. So I really think this is a barrier to entry to Python 3 that we could do without - it's the only reason I do all of my Python work in 2.x, I would jump at migrating to Python 3 if it was easily available on Fedora. Is there a solution to this that anyone knows of? Has Zed jumped to conclusions? Have I? I think he has a very valid point. I've been arguing quite a few times here that e.g. the stupid splitting up of python and python-dev packages that a great deal of people trip over should go away. But usually people here seem to think that other package management systems are the way to go, and python itself must integrate with them. E.g. providing dependency information compatible to them and their policies. I think that's bonkers. You can't support every new kid on the block claiming to be the shizzle in package management. Or the next distro with it's own packaging policies. And of course the overall release planning that says we use that ancient stable version not supported for years anymore, because it's true tested for us. IMHO the solution to this is the way Apple does it: they have a System Python. Don't mess with it. Seriously. Don't. But you can install as many other Python versions as you want, or even bundle one with your own app that depends on it. People object to this usually for two reasons: - additional waste of disk-space. Seriously? A thorough visit of youporn.com most probably fills your browser cache with more data than all possibly python installations ever can. - security issues through aged libraries. Certainly a valid point, but then this problem is not limited to Python and needs a more universal solution: Meta-information gathering about binary versions of libraries, and (safe) upgrades for these. Maybe. I haven't given much thought to this, but I think it's an OS thing more than a package distro thing. So, in summary, I think if anything, Python should liberate itself from the reigns of distro package management, and fix whatever issues there are with setuptools (or distutils or pip or distribute or whatever the cool kids use these days). And then make people use that to work with Python-packages, potentially even in individual, isolated VirtualEnvs because of package version conflicts. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Python in Linux - barrier to Python 3.x
David Cournapeau courn...@gmail.com writes: On Tue, Sep 21, 2010 at 10:23 PM, Diez B. Roggisch de...@web.de wrote: Ant ant...@gmail.com writes: Hi all, I've just seen this: http://sheddingbikes.com/posts/1285063820.html Whatever you think of Zed Shaw (author of the Mongrel Ruby server and relatively recent Python convert), he has a very good point in this. I run Fedora 12 on my home computers, and find it far too much hassle to try to get Python 3 installed. Even the 2.x's are behind - IIRC think it currently uses 2.5. So I really think this is a barrier to entry to Python 3 that we could do without - it's the only reason I do all of my Python work in 2.x, I would jump at migrating to Python 3 if it was easily available on Fedora. Is there a solution to this that anyone knows of? Has Zed jumped to conclusions? Have I? I think he has a very valid point. I've been arguing quite a few times here that e.g. the stupid splitting up of python and python-dev packages that a great deal of people trip over should go away. It is not stupid, it makes a lot of sense when you know the distributions in question. It means you have a consistent behavior independently of the language. So of course if you don't care about the rest of the ecosystem, you will think it is useless overhead. The point is that the distro doesn't care about the python eco system. Which is what I care about, and a lot of people who want to ship software. Don't get me wrong: I'm a Linux user for way over a decade, I enjoy the package management in providing a consistent distribution. What I'm talking about here are 3rd-party developers/companies/whatever, and people who want to install software that requires recent versions of packages *not* provided by their distros. I should have made that point clearer I guess. Also, I doubt that the issue is python vs python-dev - of course, given that the exact issues are not explained, we can only play guess games. The problems explained are simply outdated and crippled python versions. And to me, a python version installed that has not the distutils module is *crippled*. You can rationalize that as much as you want through some package philosophy saying we don't ship development related files, but to me a simple installation instruction that says run 'python setup.py install' which fails because of such a (debatable) decision sucks. Yes, there are corner-cases when you need GCC to compile an extension. But that's still catering to the 80% or even more (I'm guessing here) of pure-python packages. Of course, in a ideal world, distutils would hook into the distros dependency system + simply say please install python-dev first. But I'm not convinced that putting the weight here on the shoulders of the python-communtiy to deal with arbirtray decisions of the dozen or how many distros + packaging schemes out there is possible - and helpful. But usually people here seem to think that other package management systems are the way to go, and python itself must integrate with them. E.g. providing dependency information compatible to them and their policies. I think that's bonkers. You can't support every new kid on the block claiming to be the shizzle in package management. Or the next distro with it's own packaging policies. And of course the overall release planning that says we use that ancient stable version not supported for years anymore, because it's true tested for us. IMHO the solution to this is the way Apple does it: they have a System Python. Don't mess with it. Seriously. Don't. Apple's python have caused more issues than all distributions altogether for Numpy and scipy, at least as far as python itself is concerned. It is very confusing for many end-users. Exactly. My point is that I can safely install a second version besides it, and don't use the system's python that is there and kept stable for the system's own belongings. So, in summary, I think if anything, Python should liberate itself from the reigns of distro package management, and fix whatever issues there are with setuptools (or distutils or pip or distribute or whatever the cool kids use these days). And then make people use that to work with Python-packages, potentially even in individual, isolated VirtualEnvs because of package version conflicts. This kind of thinking mostly shows a poor understanding of complex deployment scenario. If everybody worked like that, you would quickly be unable to build anything stable. True, conflicts are sometimes unavoidable, but if every library keeps changing your only solution is isolated environments, you quickly have a mess of a system where many combinations of libraries are not possible. And now you are already in a mess of a system where many combinations of libraries are not possible. And the thinking of one version set to rule them all shows a poor understanding of the need for legacy code's specific version
Re: Python in Linux - barrier to Python 3.x
Antoine Pitrou solip...@pitrou.net writes: On Tue, 21 Sep 2010 17:59:27 +0200 de...@web.de (Diez B. Roggisch) wrote: The problems explained are simply outdated and crippled python versions. And to me, a python version installed that has not the distutils module is *crippled*. You can rationalize that as much as you want through some package philosophy saying we don't ship development related files, but to me a simple installation instruction that says comp.lang.python doesn't handle Linux packaging, so why don't you complain to your distro instead? Ranting on this group has zero chance of fixing the problem. comp.lang.python frequently deals with problems caused by this and other distro-related issues. Your are welcome to not participate on these discussions. The state of affairs isn't ideal, and there is improvement options on all sides. I'm just astonished that people seem to think that distros in general are better and should be left alone, than what a more python centric solution could be. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: creating python daemon ?
vineet daniel vineetdan...@gmail.com writes:
Hi
I have succesfully created daemon with python script and as next step
I am trying to give input to that python script daemon from Apache
Logshere I have got stuck and I have even checked IRC python
channel for solution. Apache is able to call the file but fails to
execute it properly and I get this error continuosly :
piped log program '/var/www/html/test.py' failed unexpectedly
How do I rectify the above error and make adjustment to the code so
that it takes input from apache.
code that I am using is as follows :
#! /usr/bin/env python
import sys,os
pid = os.fork()
os.chdir(/var/www/html/)
os.setsid()
os.umask(0)
#this line will read apache log in real time as redirected from the
CustomLog directive of Apache.
log = sys.stdin.readlines()
f = open('/var/www/logmongo.txt','a') # append log to this text file
f.write(log)
I'd appreciate if anybody could share the code that they used for
daemon or used with Apache CustomLog directive.
The above code looks errornous - you don't check for the return-value of
PID take appropriate action. Like
There is a daemonization recipe on active-state, which works nicely for
me.
http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python in Linux - barrier to Python 3.x
David Cournapeau courn...@gmail.com writes: I don't deny them their experience. Do you deny the experience of other people with *other* needs? As I already said: I don't propose to ditch the package management. I'm all fine with a distro that carefully selects it's packages and dependencies. In your previous email, you were suggesting that we should make people use a specific set of python-specific tools. That does not sound very consistent with the idea of letting people choose what they want to use. FWIW, I think those tools are already pushed too aggressively, confusing many people who use pip, virtualenv, etc... for dubious reasons (I read somewhere that I should use this), and causing numerous bug reports on the numpy/scipy mailing lists. What I suggested was that there is a python-centric solution for managing dependencies for users of Linux, Windows and OSX alike. And which offers recent versions of python to anybody. A lot of wishful thinking, admittedly. But less than trying to deal with *all* the diffences in code, style and politics of various distributions. I was not suggesting that this solution itself be manifold. The sad truth is that there currently seem to be various attempts to improve or even fix perceived or real shortcomings of of distutils or probably even more setuptools, and this is from an outside perspective a waste. But then, the survival of the fittest, so to speak, requires the death of some that are unfit. It's hard to say which approach will win. So we seem to be stuck with that at least for a while. zc.buildout, btw, seems to be going into the general direction of doing a lot (if not everything) itself. Including complete 3rd-party-packages and their builds. http://pypi.python.org/pypi/zc.buildout#buildout-examples For historic reasons I personally haven't used it yet. But it seems to scratch an itch, don't you think? Regarding the dubiousness of these reasons - I'm happy if you don't feel the pain. Good for you. I do, and frankly virtualenv is a life-saver for me in many situations. I wish it was part of core python, to create isolated environments. It sure is better than the Java-way of relying on environment-variables or giant sized commandline argument lists to specify specific version sets. However, *both* solutions cater to the obvious need of something other than pre-packaged versions in the distro. Is that such an abnorm wish? Amazing. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Python in Linux - barrier to Python 3.x
Ned Deily n...@acm.org writes: In article 87zkvbytnk@web.de, de...@web.de (Diez B. Roggisch) wrote: The point is that the distro doesn't care about the python eco system. Which is what I care about, and a lot of people who want to ship software. I don't think that is totally accurate or fair. There is regular participation in the python-dev group by packagers from various distros. For example, Matthias Klose is not only the primary Debian Python maintainer, he is also has commit privileges for Python itself and he regularly contributes patches. Currently, I see current Python 2.6.6 and 3.1.2 packages in Debian testing with current Python 2.7 and Python 3.2 alpha coming along in Debian experimental. I'm sorry, this was worded stronger than appropriate. Let me rephrase: The distros have their own (perfectly reasonable) agenda. Yet this may still conflict with the needs of users regarding e.g. contemporary package availability. I already mentioned in another post that the current debian stable features TurboGears 1.0.4. Which is by itself a problem, but also ties a lot of dependencies to ancient versions. So frankly, if I want to run (which in fact I do) a perfecly fine TurboGears2 system on lenny, I'm *forced* to use virtualenv and consorts. In other words: I think that the goals of a linux distribution don't necessarily are the same than those of a python package maintainer. In an ideal world, they would be congruent. But they aren't. My wish would be that unless that this congruency is achieved (which isn't feasible I fear), a python-only package management solution can be implemented and be adopted even by the distros without neglecting their own issues. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Encoding problem - or bug in couchdb-0.8-py2.7.egg??
Ian Hobson i...@ianhobson.co.uk writes:
Hi all,
I have hit a problem and I don't know enough about python to diagnose
things further. Trying to use couchDB from Python. This script:-
# coding=utf8
import couchdb
from couchdb.client import Server
server = Server()
dbName = 'python-tests'
try:
db = server.create(dbName)
except couchdb.PreconditionFailed:
del server[dbName]
db = server.create(dbName)
doc_id, doc_rev = db.save({'type': 'Person', 'name': 'John Doe'})
Gives this traceback:-
D:\work\C-U-Bpython tes1.py
Traceback (most recent call last):
File tes1.py, line 11, in module
doc_id, doc_rev = db.save({'type': 'Person', 'name': 'John Doe'})
File
C:\Python27\lib\site-packages\couchdb-0.8-py2.7.egg\couchdb\client.py,
line 407, in save
_, _, data = func(body=doc, **options)
File
C:\Python27\lib\site-packages\couchdb-0.8-py2.7.egg\couchdb\http.py,
line 399, in post_json
status, headers, data = self.post(*a, **k)
File
C:\Python27\lib\site-packages\couchdb-0.8-py2.7.egg\couchdb\http.py,
line 381, in post
**params)
File
C:\Python27\lib\site-packages\couchdb-0.8-py2.7.egg\couchdb\http.py,
line 419, in _request
credentials=self.credentials)
File
C:\Python27\lib\site-packages\couchdb-0.8-py2.7.egg\couchdb\http.py,
line 310, in request
raise ServerError((status, error))
couchdb.http.ServerError: (400, ('bad_request', 'invalid UTF-8 JSON'))
D:\work\C-U-B
Why? I've tried adding u to the strings, and removing the # coding
line, and I still get the same error.
Sounds cargo-cultish. I suggest you read the python introduction on
unicode.
http://docs.python.org/howto/unicode.html
For your actual problem, I have difficulties seeing how it can happen
with the above data - frankly because there is nothing outside the
ascii-range of data, so there is no reason why anything could be wrong
encoded.
But googling the error-message reveals that there seem to be totally
unrelated reasons for this:
http://sindro.me/2010/4/3/couchdb-invalid-utf8-json
Maybe using something like tcpmon or ethereal to capture the actual
HTTP-request helps to see where the issue comes from.
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python Monitoring
Glazner yoavglaz...@gmail.com writes: Hi, I'm will be writing a distributed program with parallel python and i would like to if there are any good monitoring utilities for python. I would like each remote server to post messages and to see the messages in a web-broweser or such. I googled python monitoring and found pymon but it seems a bit outdated. It's not exactly what you asked for, but the supervisord written in Python is not only capable of watching a process (admittedly always on the machine it is supposed to run), but can also be configured to have XMLRPC based (and I think even other) interfaces that a central server could poll see the status of specific processes. I admit though that it needs more configuration. Just wanted to mention it, because process management might be on your list as well. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: match pattern
rudikk00 rudik...@yahoo.com writes: I remember perl has a match function =~/H/ -- which searches if there is H pattern in line. Is there a reasonable analog of it in python? It's called a regular expression, which can be matched or searched in a string. Take a look at the module re in python. http://docs.python.org/library/re.html Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Monitoring
Glazner yoavglaz...@gmail.com writes: On Sep 20, 6:03 pm, de...@web.de (Diez B. Roggisch) wrote: Glazner yoavglaz...@gmail.com writes: Hi, I'm will be writing a distributed program with parallel python and i would like to if there are any good monitoring utilities for python. I would like each remote server to post messages and to see the messages in a web-broweser or such. I googled python monitoring and found pymon but it seems a bit outdated. It's not exactly what you asked for, but the supervisord written in Python is not only capable of watching a process (admittedly always on the machine it is supposed to run), but can also be configured to have XMLRPC based (and I think even other) interfaces that a central server could poll see the status of specific processes. I admit though that it needs more configuration. Just wanted to mention it, because process management might be on your list as well. Diez mmm... , windows not supported :( I need windows (cross platform is important) Ah. Poor you. Well, there is also pyro. It should run on all OS, and it comes with a name-service. With this, it's actually possible and probably quite easy to have all your respective clients/workers/agents connect themselves to central monitoring instance, and posting more or less accurately their state. The configuration needed should be minimal, and depending on your network setup even trivial because there is some broadcasting detection ala bonjour available. The worst that can happen is that you have to spread knowledge of a specific pyro server through a string like pyro://host/service_name or some such. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: visual studio 2010 question
David Cournapeau courn...@gmail.com writes: On Mon, Sep 20, 2010 at 3:08 PM, Ralf Haring har...@preypacer.com wrote: After running into the error Setup script exited with error: Unable to find vcvarsall.bat when trying to use easy_install / setuptools a little digging showed that the MS compiler files in distutils only support up to Studio 2008. Does anyone know if there is a timetable for when Studio 2010 will be supported? I am using python 2.6.5, but web searching seemed to show that 2.7 or 3.X didn't support it yet either. You should use VS 2008 - the lack of distutils support is only the first of a long list of issues if you want to use another compiler to build python extensions. Unless you really know what you are doing, you are better off with VS 2008, Forgive my ignorance, but AFAIK mingw can be used to build extensions for standard python distributions. After all, it's C we're talking here, not C++. So I have difficulties imagining VS2010 is that much of a problem. So - which problems you expect? Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Accessing windoze file attributes
Douglas mumm...@gmail.com writes: Hi, can anyone direct a relative newbie to the best source of info? I am writing my own backup app in Python 2.5.2 (all my company will allow me to use) using IDLE. I intend to run this app daily via the Task Scheduler to back up a mission-critical spreadsheet that only I use. It works well enough, but now I want to make it sensitive to the A attribute (only backup the file if it is set), and update it afterwards (unset it). This will help me avoid wasted disk due to needless backups. I have searched the inter-web and not found any info on how to do access/alter Windows file attributes. Please can someone direct me to a web page with suitable info (preferably with code snyppyts)? No windows nowhere here, but os.stat should be your friend. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: help with calling a static method in a private class
lallous lall...@lgwm.org writes:
How can I keep the class private and have the following work:
[code]
class __internal_class(object):
@staticmethod
def meth1(s):
print meth1:, s
@staticmethod
def meth2(s):
print meth2:,
__internal_class.meth1(s)
x = __internal_class()
x.meth2('sdf')
[/code]
By not using a double underscore. It is effectless on classes anyway
(they are not hidden because of that).
And additionally, but simply not using staticmethods at all. It's a
rather obscure feature of python - usually, classmethods are what is
considered a static method in other languages. And with that, even your
double underscores work:
class __internal_class(object):
@classmethod
def meth1(cls, s):
print meth1:, s
@classmethod
def meth2(cls, s):
print meth2:,
cls.meth1(s)
x = __internal_class()
x.meth2('sdf')
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: Distribute Non Library
narke narkewo...@gmail.com writes: My simple tool writing in python get bigger and bigger and I think I'd better split my code into several files. But, unlike what in some other languages, there is no way to compile these several files into a single executable. Before I splitting my simple tool program, I just put it in /usr/local/bin and run, very simple. Now I have to consider distribute a lot of files in a whole. I know distutils can help, but I feel it is a little uncomfortable, since I am not sharing a library, I am not sharing any thing. I just want to refactory my code. Is there a better solution to my case? Can I simply create a directory in /usr/local/bin and put my script and other used files into the directory? Does distutils help in this case? Consider using setuptools + console entry points. With these, you will automatically generate a shell-script to execute for your otherwise eggified and easy installable package. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Distribute Non Library
narke narkewo...@gmail.com writes: On 2010-09-12, Glazner yoavglaz...@gmail.com wrote: On Sep 12, 5:10 am, narke narkewo...@gmail.com wrote: My simple tool writing in python get bigger and bigger and I think I'd better split my code into several files. But, unlike what in some other languages, there is no way to compile these several files into a single executable. Before I splitting my simple tool program, I just put it in /usr/local/bin and run, very simple. Now I have to consider distribute a lot of files in a whole. I know distutils can help, but I feel it is a little uncomfortable, since I am not sharing a library, I am not sharing any thing. I just want to refactory my code. Is there a better solution to my case? Can I simply create a directory in /usr/local/bin and put my script and other used files into the directory? Does distutils help in this case? Thanks in advance. try : python mayApp.zip myApp.zip -- all your files + a __main__.py file as a starting point... looks also not decent :( i want my tool appear as an executabe, not an zip. but thank you anyway. Again: http://www.5dollarwhitebox.org/drupal/node/75 Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: What the \xc2\xa0 ?!!
Brian D brianden...@gmail.com writes:
In an HTML page that I'm scraping using urllib2, a \xc2\xa0
bytestring appears.
The page's charset = utf-8, and the Chrome browser I'm using displays
the characters as a space.
The page requires authentication:
https://www.nolaready.info/myalertlog.php
When I try to concatenate strings containing the bytestring, Python
chokes because it refuses to coerce the bytestring into ascii.
wfile.write('|'.join(valueList))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
163: ordinal not in range(128)
In searching for help with this issue, I've learned that the
bytestring *might* represent a non-breaking space.
It in fact does.
When I scrape the page using urllib2, however, the characters print
as   in a Windows command prompt (though I wouldn't be surprised if
this is some erroneous attempt by the antiquated command window to
handle something it doesn't understand).
Yes, it's trying to interpret that as two cp1252 (or whatever) bytes
instead of one unbreakable space.
If I use IDLE to attempt to decode the single byte referenced in the
error message, and convert it into UTF-8, another error message is
generated:
weird = unicode('\xc2', 'utf-8')
Traceback (most recent call last):
File pyshell#72, line 1, in module
weird = unicode('\xc2', 'utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc2 in position 0:
unexpected end of data
Which is to be expected, as you ripped a UTF-8 escape sequence in half.
If I attempt to decode the full bytestring, I don't obtain a human-
readable string (expecting, perhaps, a non-breaking space):
You obtain a non-breakable space. What do you think it should look like
in your terminal? It looks like ... nothing. Because it looks like a
space.
weird = unicode('\xc2\xa0', 'utf-8')
par = ' - '.join(['This is', weird])
par
u'This is - \xa0'
I suspect that the bytestring isn't UTF-8, but what is it? Latin1?
No, it is UTF-8
weirder = unicode('\xc2\xa0', 'latin1')
weirder
u'\xc2\xa0'
'This just gets ' + weirder
u'This just gets \xc2\xa0'
Or is it a Microsoft bytestring?
This is not weird, this is the python interpreter giving you the
representation of a unicode-object when you do not print, so you can see
what it looks like.
And because you wrongly decoded it as latin1, it's garbage anyway.
weirder = unicode('\xc2\xa0', 'mbcs')
'This just gets ' + weirder
u'This just gets \xc2\xa0'
None of these codecs seem to work.
UTF-8 worked just fine.
Back to the original purpose, as I'm scraping the page, I'm storing
the field/value pair in a dictionary with each iteration through table
elements on the page. This is all fine, until a value is found that
contains the offending bytestring. I have attempted to coerce all
value strings into an encoding, but Python doesn't seem to like that
when the string is already Unicode:
valuesDict[fieldString] = unicode(value, 'UTF-8')
TypeError: decoding Unicode is not supported
The solution I've arrived at is to specify the encoding for value
strings both when reading and writing value strings.
for k, v in valuesDict.iteritems():
valuePair = ':'.join([k, v.encode('UTF-8')])
[snip] ...
wfile.write('|'.join(valueList))
I'm not sure I have a question, but does this sound familiar to any
Unicode experts out there?
How should I handle these odd bytestring values? Am I doing it
correctly, or what could I improve?
The overall solution is to decode the page or parts of it in whatever
decoding it is delivered. You mentioned that the page is delivered in
UTF-8, so you should use whatever gives you that information to decode
the returned body.
Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: formatted input
Bob roberto.pagli...@gmail.com writes: Hi All, I have another question about formatted input. Suppose I am reading a text file, and that I want it to be something like this word11 = num11, word12 = num12, word13 = num13 etc... word21 = num21, word22 = num12, word23 = num23 etc... etc... where wordx1 belongs to a certain dictionary of words, say dic1, while wordx2 belongs to dic2, the numbers within some range and so on. I was wondering if there is something in the standard library I may use to check whether the file I am reading has a correct syntax according to my rules/dictionaries instead of implementing my own routine that would look like (pseudocode) for each line put words into a list check condition for each word No, there is no such thing. Either write something from scratch using string methods, or use pyparsing which is certainly up to the task (and much more) Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Printing forms and labels in Python
Monte Milanuk memila...@gmail.com wrote: On 6/13/10 8:00 AM, Joel Goldstick wrote: Why not go the other direction. Use python to do your processing, and send the results to excel. There are python modules that read and write excel files. Well... partly because Excel is not exactly cross-platform. Granted, the mass majority of people using a program like I have in mind will be doing so on Windows, but I was hoping to *not* tie things to any one platform if I could avoid it. Probably an overly ambitious goal at this point. The other part... maybe its my relative newbie experience level showing here, but having to call/open Excel to print something from a python app feels a little like admitting that python can't do it in a reasonable manner. If I wanted to have to open Excel to print out the results, I think I'd just as soon spend the effort and make the whole thing an Excel/VBA app and skip python entirely. Perhaps (probably) I'm not understanding some of how the lower level systems stuff works here... I thought the print drivers were there to take care of the low-level device-specific communication, and provide a somewhat simplified common interface for regular programs like M$ Word, Open Office, Adobe Reader, etc. to use. Those programs don't access the hardware directly, do they? Don't they hand off the print job to the OS-specific print drivers, and let them handle the spooling, etc.? From my (admittedly limited) understanding of things on Windows, Mac Linux, that seems to be somewhat common between platforms. Isn't there some sort of module in Python to do the same? I could understand it needing to be somewhat *platform* specific, perhaps... but I have old copies of M$ Office 2003 printing out just duckily on printers that only came out on the market less than a year ago. I updated the printer drivers and changed the default printer, but nothing printer-related in Word or Excel changed (that I know of)... but it still manages to hand off to the operating system print drivers just fine... which is kind of what I figured Python would do. Is that (to some degree) what ReportLab does? No. Reportlab just generates PDF files which your operating system will print. If you know how to - there used to be times when you just needed to copy a file to a printer device to print it. However, the overall problem here is that printer APIs are very different between os and they aren't abstracted in python to some common module. They need access to GUI libraries which python doesn't expose out of the box. So as sad as it may sound - going through some native program such as excel or acrobat to perform the printing might be your best bet. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: can't get python 2.5.5 to work on mac os x 10.4.11
new2Cocos bchk...@gmail.com wrote: Hi everyone, I posted this in the cocos2d and pyglet discussion group, I thought I'll get a response right away since my problem is quite general but I got no response. I hope you will help me!!! this is the original post http://groups.google.com/group/cocos-discuss/browse_thread/thread/f8bc92498dbd48af# best regards You Need a python framework build. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: How to automate accessor definition?
kj no.em...@please.post wrote: I need to create a class solely for the purpose of encapsulating a large number of disparate data items. At the moment I have no plans for any methods for this class other than the bazillion accessors required to access these various instance variables. (In case it matters, this class is meant to be a private helper class internal to a module, and it won't be subclassed.) What is best practice for implementing this sort of class *succinctly* (i.e. without a lot of repetitive accessor code)? Also, one more question concerning syntax. Suppose that i represents an instance of this class. Is it possible to define the class to support this syntax val = i.field i.field += 6 ...rather than this one val = i.get_field() i.set_field(i.get_field() + 6) ? You don't. Python is not Java. So just use instance attributes, and if you need bhavior when accessing an attribute, introduce a property. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: running a program on many processors
Am 08.03.10 01:18, schrieb Paweł Banyś: Hello, I have already read about Python and multiprocessing which allows using many processors. The idea is to split a program into separate tasks and run each of them on a separate processor. However I want to run a Python program doing a single simple task on many processors so that their cumulative power is available to the program as if there was one huge CPU instead of many separate ones. Is it possible? How can it be achieved? That's impossible to answer without knowing anything about your actual task. Not everything is parallelizable, or algorithms suffer from penalties if parallelization is overdone. So in essence, what you've read already covers it: if your simple task is dividable in several, independent sub-tasks that don't need serialization, multiprocessing is your friend. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: python SUDS library
Am 04.03.10 06:23, schrieb yamamoto: Hi, I tried to make a simple script with SUD library for caching torrent files. it doenst work! [versions] suds: 0.4, python: 2.6.4 [code] from suds.client import Client import base64 path = 'sample.torrent' doc = open(path, 'rb').read() encoded_doc = base64.b64encode(doc) url = 'http://torrage.com/api/torrage.wsdl' client = Client(url, cache=None) print client hash = client.service.cacheTorrent(encoded_doc) print hash [result] Suds ( https://fedorahosted.org/suds/ ) version: 0.4 (beta) build: R663-20100303 Service ( CacheService ) tns=urn:Torrage Prefixes (0) Ports (1): (CachePort) Methods (1): cacheTorrent(xs:string torrent, ) Types (0): Traceback (most recent call last): File C:\Documents and Settings\yamamoto\workspace\python\console \extract.py, line 13, inmodule result = client.service.cacheTorrent(encoded_doc) File C:\Python26\lib\site-packages\suds\client.py, line 539, in __call__ return client.invoke(args, kwargs) File C:\Python26\lib\site-packages\suds\client.py, line 598, in invoke result = self.send(msg) File C:\Python26\lib\site-packages\suds\client.py, line 627, in send result = self.succeeded(binding, reply.message) File C:\Python26\lib\site-packages\suds\client.py, line 659, in succeeded r, p = binding.get_reply(self.method, reply) File C:\Python26\lib\site-packages\suds\bindings\binding.py, line 143, in get_reply replyroot = sax.parse(string=reply) File C:\Python26\lib\site-packages\suds\sax\parser.py, line 136, in parse sax.parse(source) File C:\Python26\lib\xml\sax\expatreader.py, line 107, in parse xmlreader.IncrementalParser.parse(self, source) File C:\Python26\lib\xml\sax\xmlreader.py, line 123, in parse self.feed(buffer) File C:\Python26\lib\xml\sax\expatreader.py, line 211, in feed self._err_handler.fatalError(exc) File C:\Python26\lib\xml\sax\handler.py, line 38, in fatalError raise exception xml.sax._exceptions.SAXParseException:unknown:2:0: junk after document element this is a php code provided by torrage ? $client = new SoapClient(http://torrage.com/api/ torrage.wsdl); $infoHash = $client- cacheTorrent(base64_encode(file_get_contents(my.torrent))); ? any suggestions? Looks like the XML is malformed. Can you post it? You can insert print-statements into the library to do so, eg in line 143 of suds.bindings.binding. Diez Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: conditional import into global namespace
Am 02.03.10 21:41, schrieb mk: Jerry Hill wrote: Just import subprocess at the top of your module. If subprocess hasn't been imported yet, it will be imported when your module is loaded. If it's already been imported, your module will use the cached version that's already been imported. In other words, it sounds like Python already does what you want. You don't need to do anything special. Oh, thanks! Hmm it's different than dealing with packages I guess -- IIRC, in packages only code in package's __init__.py was executed? I don't understand this. But there is no difference regarding caching execution between packages and modules. Importing a package will execute the __init__.py, yes. But only once. As will importing modules execute them, but only once. All subsequent imports will just return the created module-object. Unless you import something under a new name! That is, you can alter the sys.path and then import a package/module under a different name - python won't detect that. Simplest example ist this: test.py class Foo(object): pass if __name__ == __main__: import test # import ourselves # this holds, because we have # __main__.Foo and test.Foo assert Foo is not test.Foo test.py However, unless you mess with python, this is none of your concern. Diez -- http://mail.python.org/mailman/listinfo/python-list
