Multiprocessing.Pipe in a daemon
So, I'm running into a somewhat crazy bug.
I am running several workers using multiprocessing to handle gearman
requests. I'm using pipes to funnel log messages from each of the
workers back to the controlling process, which iterates over the other
end of the pipe, looking for messages with poll(), and logging them
using the parent's log handler.
This works really well when I'm not daemonizing the entire thing.
However, when I daemonize the process (which happens well prior to any
setup of the pipes multiprocess.Process'es), a pipe which has
nothing in it return True for the poll(), and then blocks on the
pipe.recv() command. The gearman workers are still operational and
responsive, and only starting one worker resolves the problem.
Has anybody seen anything like this?
#Trim
# Create all of the end points
endpoints = []
log_pipes = []
for w_num in range(5):
(recv, snd) = multiprocessing.Pipe(False)
# Start the worker
logger.debug(Creating the worker {0}.format(w_num))
worker = Worker(w_num, job_name, gm_servers, snd)
# Add the worker to the list of endpoints so it can be started
endpoints.append(worker)
log_pipes.append(recv)
# Trim starting endpoints
try:
while True:
time.sleep(1)
pipe_logger(logger, log_pipes)
except (KeyboardInterrupt, SignalQuit):
pass
# Trim cleanup
def pipe_logger(logger_obj, pipes):
done =
False
while not done:
done =
True
for p in
pipes:
if p.poll(): # -- Returning true after a previous
pipe actually had data
try:
log_level, log_msg = p.recv() # -- Hanging
here
except EOFError:
continue
done = False
--
http://mail.python.org/mailman/listinfo/python-list
Closures in metaclasses
I'm running into an issue with closures in metaclasses - that is, if I
create a function with a closure in a metaclass, the closure appears
to be lost when I access the final class. I end up getting the text
'param' instead of the actual tags I am expecting:
ALL_TAGS = ['a', 'abbr', 'acronym', 'address', 'applet', 'b', 'bdo',
'big'] #snip
def _tag_meta(name, bases, dict_):
for tag in ALL_TAGS:
def generic_tag(*args, **kwargs):
return Tag._generate_tag(tag, *args, **kwargs)
#generic_tag = eval(lambda *args, **kwargs: Tag._generate_tag
('%s', *args, **kwargs) % tag)
dict_[tag] = staticmethod(generic_tag)
return type(name, bases, dict_)
class Tag(object):
__metaclass__ = _tag_meta
@staticmethod
def _generate_tag(tag_name, *args, **kwargs):
# Does the expected, following is just for the example's sake
return tag_name
Here's the output from my actual class:
$ python
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type help, copyright, credits or license for more information.
from htmlgen import Tag
Tag.html(foo)
'param\nfoo\n/param'
Using the eval shown above works as expected, but I'd rather go with
the closures, if someone can help me figure out what's going on.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Closures in metaclasses
On Jan 21, 11:24 am, Arnaud Delobelle arno...@googlemail.com wrote:
Falcolas garri...@gmail.com writes:
I'm running into an issue with closures in metaclasses - that is, if I
create a function with a closure in a metaclass, the closure appears
to be lost when I access the final class. I end up getting the text
'param' instead of the actual tags I am expecting:
ALL_TAGS = ['a', 'abbr', 'acronym', 'address', 'applet', 'b', 'bdo',
'big'] #snip
def _tag_meta(name, bases, dict_):
for tag in ALL_TAGS:
def generic_tag(*args, **kwargs):
return Tag._generate_tag(tag, *args, **kwargs)
#generic_tag = eval(lambda *args, **kwargs: Tag._generate_tag
('%s', *args, **kwargs) % tag)
dict_[tag] = staticmethod(generic_tag)
return type(name, bases, dict_)
This is almost a FAQ and has nothing to do with metaclasses. The
simplest solution usually involves adding a default argument 'tag=tag'
to the function you define (here, generic_tag), but you can't do this
here because you have a **kwargs argument. Instead, you can use a
closure and do this for example:
def factory(tag):
def generic_tag(*args, **kwargs):
return Tag._generate_tag(tag, *args, **kwargs)
return generic_tag
def _tag_meta(name, bases, dict_):
for tag in ALL_TAGS:
dict_[tag] = staticmethod(factory(tag))
return type(name, bases, dict_)
I see - I was thinking it would preserve the closure from the for
statement, but I can see now why that would be wrong.
However, I am usure about why you are using a metaclass.
HTH
--
Arnaud
It was the easiest way I found to add a lot of static methods to the
Tag class without writing each one out. __getattr__ was not working
for this application. This is for a very simple application, and I
didn't want to add a lot of complexity to it's use. I'm always open
for other options OTOH.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Closures in metaclasses
On Jan 21, 12:10 pm, Arnaud Delobelle arno...@googlemail.com wrote: On Jan 21, 6:37 pm, Falcolas garri...@gmail.com wrote: On Jan 21, 11:24 am, Arnaud Delobelle arno...@googlemail.com wrote: It was the easiest way I found to add a lot of static methods to the Tag class without writing each one out. __getattr__ was not working for this application. This is for a very simple application, and I didn't want to add a lot of complexity to it's use. I'm always open for other options OTOH. This should work (untested): class Tag(object): @staticmethod def _generate_tag(tag_name, *args, **kwargs): # Does the expected, following is just for the example's sake return tag for tag in ALL_TAGS: setattr(Tag, tag, staticmethod(factory(tag))) Or you could override __getattr__ -- Arnaud I tried overriding __getattr__ and got an error at runtime (the instance did not have xyz key, etc), and the Tag dict is not modifiable (granted, I tried Tag.__dict__[tag] = spam, not setattr()). -- http://mail.python.org/mailman/listinfo/python-list
Re: Closures in metaclasses
On Jan 21, 1:55 pm, Peter Otten __pete...@web.de wrote: Falcolas wrote: I tried overriding __getattr__ and got an error at runtime (the You can either move __getattr__() into the metaclass or instantiate the class. I prefer the latter. Both approaches in one example: class Tag: ... class __metaclass__(type): ... def __getattr__(self, name): return %s via metaclass % name ... def __getattr__(self, name): return %s via class % name ... Tag.yadda 'yadda via metaclass' tag = Tag() tag.yadda 'yadda via class' Very nice, thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: how to format a python source file with tools?
On Nov 30, 7:37 am, gil_johnson x7-g5w...@earthlink.net wrote: On Nov 27, 9:58 am, Diez B. Roggisch de...@nospam.web.de wrote: [...] so i would like to have a tool to intelligently format the code for me and make the code more beautiful and automated. This is not possible. Consider the following situation: [...] Both are semantically radically different, and only you know which one is the right one. Diez I have to agree with Diez, there is no way to automate this. Some human intervention is needed. What I would like is an editor that will indicate what Python will consider a logical block (and sub-block, and sub-sub-block, etc.) It's complicated. I've tried to think of a way to do it, and have gotten lost after a few changes of indentation. Does anyone know of such a thing? I miss curly braces with an editor that will highlight matching parentheses, braces, etc. Gil At least with Windows, you get a number of scripts included in your Python install - under the python directory/tools/scripts. There are several scripts here which are intended to help with indentation issues - such as reindent and pindent. Those might help you out some. Nonetheless, it would be better to implement coding standards that everyone can stick to. -- http://mail.python.org/mailman/listinfo/python-list
Re: TODO and FIXME tags
On Nov 17, 4:27 am, Martin P. Hellwig martin.hell...@dcuktec.org wrote: Ben Finney wrote: Chris Rebert c...@rebertia.com writes: 2009/11/16 Yasser Almeida Hernández pedro...@fenhi.uh.cu: How is the sintaxis for set the TODO and FIXME tags...? There is no special syntax for those. Some people use them in comments, but it's just a convention. This is true. However, the convention is fairly well established, and many text editor default configurations will highlight the strings ‘TODO’, ‘FIXME’, and others wherever they appear in comments. There's no widely-followed “syntax” for this convention, though. I have noticed that within Python XXX as an identifier is fairly popular as well - it's used throughout the standard libraries, and is recognized by many syntax highlighting schemes, as well as Pylint. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: ImportError: No module named _md5 - please help
On Oct 29, 9:13 am, user7304 wadie...@gmail.com wrote: Hi, I am trying to run a python script and got this error. import _md5 ImportError: No module named _md5 Googling the problem suggested that I install the 'py25-hashlib'. the following does not work for me 'sudo port install py25-hashlib' , trying to install MacPorts raised many problems. My question is: any idea on how to install it using yum? I am running python 2.6.2 on a centos machine. Many thanks for your help. Try using hashlib.md5. From the docs: import hashlib m = hashlib.md5() m.update(Nobody inspects) m.update( the spammish repetition) m.digest() '\xbbd\x9c\x83\xdd\x1e\xa5\xc9\xd9\xde\xc9\xa1\x8d\xf0\xff\xe9' http://www.python.org/doc/2.5.2/lib/module-hashlib.html Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: Recommended number of threads? (in CPython)
On Oct 29, 9:56 am, mk mrk...@gmail.com wrote: Hello everyone, I wrote run-of-the-mill program for concurrent execution of ssh command over a large number of hosts. (someone may ask why reinvent the wheel when there's pssh and shmux around -- I'm not happy with working details and lack of some options in either program) The program has a working queue of threads so that no more than maxthreads number are created and working at particular time. But this begs the question: what is the recommended number of threads working concurrently? If it's dependent on task, the task is: open ssh connection, execute command (then the main thread loops over the queue and if the thread is finished, it closes ssh connection and does .join() on the thread) I found that when using more than several hundred threads causes weird exceptions to be thrown *sometimes* (rarely actually, but it happens from time to time). Although that might be dependent on modules used in threads (I'm using paramiko, which is claimed to be thread safe). Since you're creating OS threads when doing this, your issue is probably more related to your OS' implementation of threads than Python. That said, several hundred threads, regardless of them being blocked by the GIL, sounds like a recipe for trouble on most machines, but as usual YMMV. If you're running into problems with a large number of connections (not related to a socket limit), you might look into doing it asynchronously - loop over a list of connections and do non-blocking reads to see if your command has completed. I've done this successfully with pexpect, and didn't run into any issues with the underlying OS. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: list comprehension problem
On Oct 29, 9:31 am, Diez B. Roggisch de...@nospam.web.de wrote: mk wrote: Hello everyone, print hosts hosts = [ s.strip() for s in hosts if s is not '' and s is not None and s is not '\n' ] print hosts ['9.156.44.227\n', '9.156.46.34 \n', '\n'] ['9.156.44.227', '9.156.46.34', ''] Why does the hosts list after list comprehension still contain '' in last position? I checked that: print hosts hosts = [ s.strip() for s in hosts if s != '' and s != None and s != '\n' ] print hosts ..works as expected: ['9.156.44.227\n', '9.156.46.34 \n', '\n'] ['9.156.44.227', '9.156.46.34'] Are there two '\n' strings in the interpreter's memory or smth so the identity check s is not '\n' does not work as expected? This is weird. I expected that at all times there is only one '\n' string in Python's cache or whatever that all labels meant by the programmer as '\n' string actually point to. Is that wrong assumption? Yes. Never use is unless you know 100% that you are talking about the same object, not just equality. Diez I'd also recommend trying the following filter, since it is identical to what you're trying to do, and will probably catch some additional edge cases without any additional effort from you. [s.strip() for s in hosts if s.strip()] This will check the results of s.strip(), and since empty strings are considered false, they will not make it into your results. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to write a daemon program to monitor symbolic links?
On Oct 23, 1:25 pm, Peng Yu pengyu...@gmail.com wrote: As far as I know, linux doesn't support a system level way to figure out all the symbolic links point to a give file. I could do a system wide search to look for any symbolic link that point to the file that I am interested in. But this will be too slow when there are many files in the systems. I'm thinking of writing a daemon program which will build a database on all the symbolic links that point to any files. Later on, whenever I change or remove any file or symbolic link, I'll will notify the daemon process the changes. By keeping this daemon process running, I can quickly figure out what symbolic links are pointing to any give file. But I have never make a daemon program like this in python. Could somebody point me what packages I need in order to make a daemon process like this? Thank you! I would recommend looking into some articles on creating well behaved daemons and review python recipes for creating daemonic processes. From there, it's mostly a matter of writing code which is fairly self reliant. The ability to write to the system logs (Python module syslog) helps quite a bit. http://www.google.com/search?q=writing+daemons http://code.activestate.com/recipes/278731/ I typically write a program which will run from the command line well, then add a switch to make it a daemon. That way, you have direct control over it while writing the daemon, but can then daemonize it (using the activestate recipe) without making changes to the code. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to write a daemon program to monitor symbolic links?
On Oct 23, 1:38 pm, Falcolas garri...@gmail.com wrote: On Oct 23, 1:25 pm, Peng Yu pengyu...@gmail.com wrote: As far as I know, linux doesn't support a system level way to figure out all the symbolic links point to a give file. I could do a system wide search to look for any symbolic link that point to the file that I am interested in. But this will be too slow when there are many files in the systems. I'm thinking of writing a daemon program which will build a database on all the symbolic links that point to any files. Later on, whenever I change or remove any file or symbolic link, I'll will notify the daemon process the changes. By keeping this daemon process running, I can quickly figure out what symbolic links are pointing to any give file. But I have never make a daemon program like this in python. Could somebody point me what packages I need in order to make a daemon process like this? Thank you! I would recommend looking into some articles on creating well behaved daemons and review python recipes for creating daemonic processes. From there, it's mostly a matter of writing code which is fairly self reliant. The ability to write to the system logs (Python module syslog) helps quite a bit. http://www.google.com/search?q=writing+daemonshttp://code.activestate.com/recipes/278731/ I typically write a program which will run from the command line well, then add a switch to make it a daemon. That way, you have direct control over it while writing the daemon, but can then daemonize it (using the activestate recipe) without making changes to the code. Garrick One other note - sorry for the double post - if you look at other programs which maintain a DB of files, such as unix' slocate program, update the DB as a daily cron job. You may want to also consider this route. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: Checking a Number for Palindromic Behavior
On Oct 20, 11:18 am, ru...@yahoo.com wrote: Why *not* answering a question in comp.lang.python because you think it is homework is BAD. 1) It may look like a homework problem to you but it probably isn't. Seehttp://groups.google.com/group/comp.lang.python/msg/8ac6db43b09fdc92 Homework comes in many forms - school driven homework should be treated the same as self driven research, IMO. You're not doing it to be told the answer, you're likely doing it to learn. 2) When you publicly accuse someone of cheating (and even asking is tantamount to an accusation unless done very tactfully), especially without anything more than your feelings to back it up, you will likely anger the poster, contribute to an generally unpleasant atmosphere in the newsgroup, and intimidate other people who want to ask legitimate questions. Arguing with a long standing Usenet and clp traditions are just as likely to cause an unpleasant atmosphere. 3) You are not responding only to the original poster; there are many other silent readers who are interested in the answer and whom you are depriving of knowledge by refusing to answer. MRAB provided a perfect answer - anybody who wants to know more, or could not connect the dots, can always ask for more information. 4) When you post a specific solution to a question, usually a number of other people will respond with alternate or better solutions. While perhaps overkill for the original poster who likely will be satisfied with any answer, such discussion greatly benefits other readers. See previous. 5) Although working out an answer oneself is the usual goal of homework problems, it is not the only way to learn. Often, when one is really stuck, one can learn what one is supposed to by seeing the fully worked out problem's answer. You, who don't know anything about the poster, are not in a position to decide for him/her what the best way of learning is. The poster is also free to ignore your answer if he/she chooses. Again, if the original, directing answer was not sufficient, the original poster is welcome to ask for more information. Doing so will likely help them as much as it would help this silent reader population. 6) Please don't apply your abstract moral standards to the entire rest of the world, knowing nothing about the particular circumstances of the poster. So, let the poster give the circumstances. We have just as much right to question his motives as you have to question ours. 7) If the poster is determined to cheat, he/she will do so with or without your help. Your self-righteous stand will serve only to generate the above undesirable results without changing the poster's behavior. How did MRAB's response negatively affect anybody? Of course, whether you choose to provide a specific answer to something you think is homework, or not, is ultimately a personal decision and you are free to follow your conscious. Just please don't demand that every other participant in this group adopt your personal standards. When you join a long standing community, you're expected to follow their conventions. Don't top post, don't do someone else's homework, etc. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: python along or bash combined with python (for manipulating files)
On Oct 13, 10:18 pm, TerryP bigboss1...@gmail.com wrote: On Oct 14, 2:13 am, Peng Yu pengyu...@gmail.com wrote: Bash is easy to use on manipulating files and directories (like change name or create links, etc) and on calling external programs. For simple functions, bash along is enough. However, bash does not support the complex functions. Python has a richer library that could provide support for complex functions (such compute the relative path between two paths). I'm wondering for a task that can not be done with bash along whether it would be better to do in pure python or with a mix of both python and bash. What I care is mostly coding speed and a little bit maintainability (but not much). Can somebody provide some experience on when to combine python and bash and when to use pure python? bash can **not** manipulate files and directories beyond things like the '' and '' I/O redirections, and some minor loading/saving of state data from/to files (command history, directory stack, etc). Most of what you refer to are **separate operating system specific programs** and have absolutely nothing to do with the shell. Very sophisticated scripts are possible using bash and ksh, there is even a form of ksh that has tk capabilities! (tksh). The Python and Bourne-derived languages are however fundamentally different creatures, and use very different data models. You should **not** write Python (or Perl) scripts as if they were shell scripts -- doing so is very bad practice. When you want a shell script, write a shell script. When you write a Python script, write a Python script. It really is that simple. As a rule of thumb, when you have need of data structures beyond what scalar strings and very simple word lists can provide -- you should use Python. bash and ksh provide support for arrays, and ksh even has dictionaries! (Hashes in Perl speak.) That makes programming in bash/ ksh more robust then pure sh, but also less portable. The best time to use bash is when you require bash specific features, other wise don't use bash. The same can be said for ksh. When the words array, dictionary, class, object, and/or using multiple source files comes to mind when implementing a program - you probably want to use Python, Perl, Ruby, or some other general programming language, not a shell scripting language like bash. You should be cautious to avoid mixing bash and Python code in one file. If maintainability is not a factor in what you are writing, then you should probably not be writing code in any language unless it is the language of Mathematics (and even then, maintainability is a wise consideration). -- TerryP. Just Another Programmer. With all of Terry's admonitions in mind, Python scripts do integrate very well as a individual tool within a shell toolchain. With the multiple command line parsers and the ease of reading stdin and writing to stdout, it's fairly trivial to make a script which integrates cleanly into a bash script (or oneliner). It's trivial to implement a script which will either work with files, or work with stdin/stdout. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: Python XMLRPC question
On Oct 13, 12:47 pm, prasanna prasa...@ix.netcom.com wrote: In using Python's XMLRPC, there is a statement that gets printed on stdout of the form: localhost - - [12/Oct/2009 23:36:12] POST /RPC2 HTTP/ 1.0 200 - Where does this message originate? Can I turn it off, or at least redirect it into a logging file? I am planning to run the server code automatically at start up and wouldn't have a terminal window open to get this message. I guess I could redirect/pipe it to a file, but it would be more useful I could send it to the same log file that I have the server writing other messages to. Thanks for any help. Looking back through the SimpleXMLRPCServer code, it appears that this happens if the logRequests parameter in the SimpleXMLRPCServer class initialization is set to True, which it is by default. The underlying implementation of the logging is in BaseHTTPServer.py, which uses sys.stderr. Looks like the simplest way to change that would be to inherit from the SimpleXMLRPCRequestHandler class and implement your own log_request method. You could then pass that to the SimpleXMLRPCServer constructor. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: The rap against while True: loops
On Oct 12, 12:32 pm, Mensanator mensana...@aol.com wrote: On Oct 12, 1:02 pm, John Reid j.r...@mail.cryst.bbk.ac.uk wrote: Mensanator wrote: On Oct 12, 3:36 am, greg g...@cosc.canterbury.ac.nz wrote: Mensanator wrote: while not done: ... if n==1: done = True ... Seems to me that 'while not done:' is no better than 'while True:', because in both cases you have to look inside the loop to find out what the exit condition is. Using a more meaningful name for the flag can help, but you can't teach someone that just by giving them an overly simplified rules such as never use while True:. They'll probably just replace it with 'while not done:' and think they've improved things, without ever really understanding the issue. You're missing the point. It's not that you have to look inside for the terminating condition. It's that you don't need a break. Nothing wrong with a having a break IMHO. My opinion is that there is everything wrong with having a break. I don't think I have ever used one, I write code that doesn't depend on that crutch. while not done: seems very dangerous to me as you'd have to del done before writing the same construct again. That's the sort of thing that leads to errors. Duh. I won't write silly code like that either. If I need more than one loop structure then I'll do something like while not done_with_this while not done_with_that Besides, since I _always_ initialize the flag before entering a loop, the flag can be reused and doesn't have to be deleted (as long as the loops aren't nested). And since I don't use goto, there's no chance the initialization can be avoided. The best way to avoid the pitfalls of spaghetti code is to not write it in the first place. How do you manage code where you need to drop out of a neatly written for or while loop early? I don't use break frequently, but just like gotos, it does have it's place in well written code. Glad to hear, by the way, that you don't use gotos in Python. =D Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: best vi / emacs python features
On Oct 8, 7:23 am, Jean-Michel Pichavant jeanmic...@sequans.com wrote: Chris Jones wrote: On Wed, Oct 07, 2009 at 07:06:08PM EDT, TerryP wrote: [..] I am a freak: I do not use nor want syntax highlighting. I don't want my editor to understand mail, irc, or the www either, I want it to edit text efficiently so I can go on with the rest of my life as soon as possible. Given the choice of using a space cadets editor like emacs or something primitive one like ed, I would choose *ed* just to speed things up and save on wrist strain. Before I read a tutorial about vi, I used XEmacs very happily---vi just lines up better with how my brain works. -- It is also general consensus that I am nuts ;) I don't think so. Always felt that syntax highlighting for instance is way overrated. Haven't tried ed yet, but since I have already stripped down my everything to what I thought was minimal, now that you mention it, this may be where I'm bound. I do have a question: You mentioned Vim's clientserver mode. What's it good for? Thanks, CJ Strange how things can differ from one person to another. Syntax highlighting is the feature I'm missing the most when I'm using pure vi. Looks like some are getting annoyed by it, I think it provides useful and meaningful informations through colors. JM I have found that syntax hilighting is useful, as long as it isn't overwhelming. Zenburn is a great highlighting scheme which is fairly easy on the eyes to use (no startling colors anywhere), yet the differences are enough for me to quickly pick out syntax elements. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: best vi / emacs python features
On Oct 7, 10:44 am, Jean-Michel Pichavant jeanmic...@sequans.com wrote: OdarR wrote: hello, * this is not a troll * which kind of help you have with your favorite editor ? personnally, I find emacs very nice, in the current state of my knowledge, when I need to reindent the code. you know how this is critical in python...:-) I don't use other python-mode features for the moment, maybe you have some advices. I used with success an Xemacs on Solaris, as well as a Aquamacs on Mac. For the vi/vim guys, explain us the best features this editor give you in your py files ! thanks, Olivier Being a vi fan, I can just tell you that emacs is for loosers, and no one will dare to challenge this. I concur 100% with this statement. That said, I like how my install of VIM 'just works' with Python files, doing a good job of coloring and auto indentation. I don't like, however, how an arbitrary colon will trigger auto indentation on the next line. I'm guessing, though, that this is just an issue with my particular syntax file. Garrick -- http://mail.python.org/mailman/listinfo/python-list
Re: Storing a C pointer in a Python class instance
On Sep 29, 2:27 am, lallous lall...@lgwm.org wrote: Hello From my C extension module I want to store a C pointer in a given PyObject. The only way I figure how to do it is to use Py_BuildValues and store the poiner casted to Py_ssize_t, thus: Py_BuildValues(n, (Py_ssize_t)my_ptr) Can it be done differently? Regards, Elias You can use a PyCObject_FromVoidPtr http://docs.python.org/c-api/cobject.html PyArg_ParseTuple(args, O, pyVoidPointer); castPointer = (type *) PyCObject_AsVoidPtr(pyVoidPointer); return PyCObject_FromVoidPtr((void *) castPointer, NULL); -- http://mail.python.org/mailman/listinfo/python-list
Re: Cannot get POST to work
On Sep 29, 10:24 am, tedpot...@gmail.com tedpot...@gmail.com
wrote:
Hi,
I'm trying to post data to a short test script in php I wrote.
The python code to do the post is
import httplib
#server address
conn = httplib.HTTPConnection(localhost)
#file location
conn.request(POST, /programming/bots/test.php,ted=fred)
r1 = conn.getresponse()
print r1.status, r1.reason
data1 = r1.read()
print data1
conn.close()
print new ok
The PHP script is
printhello br;
print $_POST[ted];
Ted post
I can't speak to what is wrong with your current script - instead I
would recommend the higher level urllib libraries:
(Untested)
import urllib2, urllib
response = urllib2.open(http://localhost/programming/bots/test.php;,
urllib.urlencode({ted: fred}))
print response.read()
response.close()
~Garrick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Podcast catcher in Python
On Sep 11, 8:20 am, Chuck galois...@gmail.com wrote: Hi all, I would like to code a simple podcast catcher in Python merely as an exercise in internet programming. I am a CS student and new to Python, but understand Java fairly well. I understand how to connect to a server with urlopen, but then I don't understand how to download the mp3, or whatever, podcast? Do I need to somehow parse the XML document? I really don't know. Any ideas? Thanks! Chuck You will first have to download the RSS XML file, then parse that file for the URL for the audio file itself. Something like eTree will help immensely in this part. You'll also have to keep track of what you've already downloaded. I'd recommend taking a look at the RSS XML yourself, so you know what it is you have to parse out, and where to find it. From there, it should be fairly easy to come up with the proper query to pull it automatically out of the XML. As a kindness to the provider, I would recommend a fairly lengthy sleep between GETs, particularly if you want to scrape their back catalog. Unfortunately, I no longer have the script I created to do just such a thing in the past, but the process is rather straightforward, once you know where to look. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Creating slice notation from string
On Sep 2, 3:55 pm, bvdp b...@mellowood.ca wrote: I'm trying to NOT create a parser to do this and I'm sure that it's easy if I could only see the light! Is it possible to take an arbitrary string in the form 1:2, 1, :-1, etc. and feed it to slice() and then apply the result to an existing list? For example, I have a normal python list. Let's say that x = [1,2,3,4] and I have a string, call it s', in the format [2:3]. All I need to do is to apply s to x just like python would do. I can, of course, convert x to a list with split(), convert the 2 and 3 to ints, and then do something like: x[a:b] ... but I'd like something more general. I think the answer is in slice() but I'm lost. Thanks. You might also consider looking at the operator module, which provides to potentially useful methods - itemgetter and getslice. Neither will directly use the string containing the bare slice notation, however they do provide another alternative to directly using eval. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: your favorite debugging tool?
On Aug 23, 1:21 am, Hendrik van Rooyen hend...@microcorp.co.za wrote: On Saturday 22 August 2009 16:49:22 Aahz wrote: In article mailman.227.1250951162.2854.python-l...@python.org, Esmail ebo...@hotmail.com wrote: What is your favorite tool to help you debug your code? I've been getting along with 'print' statements but that is getting old and somewhat cumbersome. Despite the fact that I've been using Python for more than a decade, print is still my mainstay (or possibly logging to a file). Same here, although I have not been abusing python for as long as Aahz has been using it. ... And the final arbiter is of course the interactive prompt. - Hendrik The only thing I would add to these comments is the fact that writing helper functions for using print with complicated routines is very simple, though perhaps not obvious. For example, consider the following for unrolling list comprehensions or generators without having to re-write them: def trace(iterable): for x in iterable: print x yield x ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: conditional for-statement
On Aug 25, 11:25 am, seb sdemen...@gmail.com wrote: We could as consistenly explain that the syntax for n in range(10) if n%3==0: body means for n in range(10): if n%3==0: body This syntax has also the benefit of avoiding an extra level of indentation (the one for the if) that bears no real meaning on a structural level. Maybe a PEP could do the job... Sébastien So, what part of the statement does the if statement belong to; particularly a concern considering this is valid python: for x in y if y else z: body You can always do the following at the cost of 6 symbols, and the gain of clarity: for n in (x for x in y if y%3==0): body ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: conditional for-statement
On Aug 25, 1:58 pm, seb sdemen...@gmail.com wrote:
On Aug 25, 9:42 pm, Falcolas garri...@gmail.com wrote:
On Aug 25, 11:25 am, seb sdemen...@gmail.com wrote:
So, what part of the statement does the if statement belong to;
particularly a concern considering this is valid python:
for x in y if y else z:
body
can this be done in list/set/dict comprehensions/generator
expressions ?
It's a statement, so anywhere you use a statement (such as in
generators and list comprehensions), it can exist. for ... in ... only
requires that the statement return an iterable, it doesn't matter what
statement you use to get there.
For example, the following is perfectly valid (if nonsensical - I
don't use ternary operators very often in my programming, so I don't
have many good examples of their use)
list = [x if x else y for x, y in iterable]
it is in fact precisely to avoid this sort of line:
for n in (x for x in y if x%3==0):
and have instead the (more readable IMO) line
for n in y if n%3==0:
with:
- 1 for ... in ... instead of 2 (where one is the repetition of the
other)
- no parentheses
- no extra technical variable with local binding to the expression
generator ('x')
it looks more pythonic to me but it is a personal taste.
It doesn't feel clear to me, which is why I would have to disagree.
IIRC, these very filters are the main reason that list comprehensions,
and later one-line generators and dictionary comprehensions were
originally created: so you can filter the stream before acting on it.
One statement performs one action - very precise and understandable.
It's just my two cents, but I would not agree that adding a filter
keyword to for loops is required. Not when we have the filter
function, various types of comprehensions, and if statements which all
provide that very functionality.
~G
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to launch a function at regular time intervals ?
On Aug 12, 3:09 pm, David davig...@googlemail.com wrote:
Hi all, I'm trying to launch a function at regular time intervals but
cannot find the way to do it. Here is the code I wrote (time_interval
is a user defined variable in seconds):
[snip]
Has anyone run into a similar problem (and solved it) ?
Thanks for your help
I did - as part of a script where I needed to send load into a system
at a steady rate. I ended up using threading to do the function calls,
since they were not guaranteed to complete before the next interval.
duration_start = time.time()
interval_counter = 0
while time.time() - duration_start duration:
for thread_count in xrange(numthreads):
threading.Thread(target=exec_thread, kwargs={#snip
unimportant#}).start()
interval_counter += 1
time.sleep((duration_start + (interval * interval_counter)) -
time.time())
Executing this loop with a simple echo and time print showed that
there was no creep over time, and the deviation between intervals was
around 1/100th of a second.
I'm fairly sure I'm creating some gnarly memory leaks and such by not
joining spent threads, but it's been a non-issue in my usage. Adding a
list to keep track of the threads and join on complete threads would
be fairly trivial to implement.
I think for simpler applications, using threading.Timer to kick off
the function would work just as well.
~G
--
http://mail.python.org/mailman/listinfo/python-list
Re: Confessions of a Python fanboy
On Jul 31, 3:49 am, Masklinn maskl...@masklinn.net wrote: On 31 Jul 2009, at 10:25 , Chris Rebert wrote: On Fri, Jul 31, 2009 at 1:21 AM, Xavier Hocont...@xavierho.com wrote: On Fri, Jul 31, 2009 at 6:08 PM, Masklinn maskl...@masklinn.net wrote: snip... but since Python doesn't have anonymous functions that usage tends to be a bit too verbose ... snip Sorry to interrupt, but wouldn't lambda in Python be considered as 'anonymous functions'? I believe full anonymous functions was intended by the author. lambdas are limited to a single expression. Yes, and they're limited to a single *expression*, so before Python 3, lambda: print foo is out. Likewise, it isn't possible to have an if/ else statement within a lambda (though a ternary is ok), or a with, … since Python statements aren't expressions. Perhaps you can't do lambda foo: print foo, but you *can* do lambda x: sys.stdout.write(x). Combined with the ternary if, it seem sufficient if you want to stick to the ideal Simple is better than Complex. Sure, with statements allowed in anonymous functions, you can save on a line of typing (def blargh(x):), but I don't feel the obfuscation is worth the cost. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Confessions of a Python fanboy
On Jul 29, 9:06 pm, r rt8...@gmail.com wrote: 1.) No need to use () to call a function with no arguments. Python -- obj.m2().m3() --ugly Ruby -- obj.m1.m2.m3 -- sweeet! Man, i must admit i really like this, and your code will look so much cleaner. I personally would not prefer this, and would likely continue to use (), precisely for code clarity - let me explain: foo.nextval foo.val foo.previousval Which of the calls above referenced instance variables, and which ones called functions which changed the internal state of foo? I would have trouble saying, just based on the calls above. I would have to go back to the definition or documentation of foo to identify which is doing what. On the other hand, the following gives a better clue as to what is happening (granted not perfect, but better): foo.nextval() foo.val foo.previousval() ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Confessions of a Python fanboy
On Jul 30, 11:56 am, Masklinn maskl...@masklinn.net wrote: On 30 Jul 2009, at 19:37 , Jean-Michel Pichavant wrote: r wrote: How do I know if foo.value is an attribute or if it is a method that returns the foo value ? It cannot be an attribute. Ruby doesn't give access to attributes, they're always solely private (as in Smalltalk). Also you shouldn't reply to r, he has no idea about what he's talking about. That doesn't sound like a feature I would appreciate much, though I am a big fan of we're all adults here programming. Though if they are all functions, then adding parentheses wouldn't make much difference, so offering the option of getting rid of them makes sense. I would have trouble saying, just based on the calls above. I would have to go back to the definition or documentation of foo to identify which is doing what. On the other hand, the following gives a better clue as to what is happening (granted not perfect, but better): Well... it doesn't give much of a clue no really. It, with few exceptions, tells me that fetching the value foo.val will not modify the state of foo. Important stuff, at least to me. -- http://mail.python.org/mailman/listinfo/python-list
Re: n00b confusion re: local variable referenced before assignment error
On Jun 19, 10:16 am, Wells Oliver we...@submute.net wrote: Writing a class which essentially spiders a site and saves the files locally. On a URLError exception, it sleeps for a second and tries again (on 404 it just moves on). The relevant bit of code, including the offending method: [snip] But what I am seeing is that after a retry (on catching a URLError exception), I see bunches of UnboundLocalError: local variable 'handler' referenced before assignment errors on line 38, which is the file.write(handler.read()) line.. What gives? 'Handler' is only assigned in the try statement, so if you error into your exception clause, nothing will have been bound to the name 'handler', causing the exception you're seeing. -- http://mail.python.org/mailman/listinfo/python-list
Re: What text editor is everyone using for Python
I am a long time VIM user, and I likely will not change that. The speed, ease of use and functionality, for me, is worth the time spent learning how to use it. My secondary editor on the desktop is UltraEdit, which does a fine job as a text editor and has all the same functionality of VIM - yet despite 2 years on it (they won't allow me GVIM at work), I can't get to the same level of productivity with it as I can with VIM. Ditto Eclipse... I spent more time figuring out how to get a program to run properly than coding. The limited autocomplete and function jump-list were not worth the pain of getting it working, IMO. On May 29, 2:01 am, Steven D'Aprano st...@remove-this- cybersource.com.au wrote: (1) Closed source editors have the same functional requirements as open source editors. (2) If you're waiting for perfection, you'll be waiting forever. (3) Why independent of the OS? When is the last time you've used a system without an OS? Forth programmers in the 1970s used an editor that was OS independent -- it managed files using its own unique file structure, managed memory itself, etc. Why do you want to go back there? As to your points, Steven: 1) I would agree with this point. VI and it's children were written to a different interpretation of said rules, however. They were built around the speed at which a human can use a keyboard with natural finger movements, at the expense of discoverability and intuitiveness. 2) VIM may not be perfect, but it's really darned close. ;-) 3) OS independence, IMO, is related more to the ability to use the tool on every OS, rather than on the lack of an OS. -- http://mail.python.org/mailman/listinfo/python-list
Re: Request For Comment
On Apr 7, 12:57 pm, a...@pythoncraft.com (Aahz) wrote: How RFC1 got created: http://www.nytimes.com/2009/04/07/opinion/07crocker.html Thanks for bring that forward; I thought it was an interesting read and a nice polite (yet much needed) little jab at status-quo. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Why does Python not return first line?
On Mar 15, 6:25 pm, Gilles Ganault nos...@nospam.com wrote: address = re_address.search(response) if address: address = address.group(1).strip() #Important! for item in [\t,\r, br /]: address = address.replace(item,) As you found, your script works just fine, it's just that during terminal output the \r performs a carriage return and wipes out everything prior to it. FWIW, I've rarely seen a \r by itself, even in Windows (where it's usually \r\n). Unix generally just outputs the \n, so my guess is that some other process which created the output removed newline characters, but didn't account for the carriage return characters first. Wiping out the \r characters as you did will solve your display issues, though any other code should read right past them. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Stopping SocketServer on Python 2.5
On Mar 10, 7:19 pm, David George d...@eatmyhat.co.uk wrote: So, my question is, is there any way to stop a SocketServer that's been told to server forever in python 2.5? serve_forever, in python 2.5, is simply coded as: while 1: self.handle_request() So, instead of calling serve_forever, call handle_request in your own loop with a shutdown flag. Assuming, of course, that you aren't using the Forking server, in which case things get more interesting. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Stopping SocketServer on Python 2.5
On Mar 11, 12:28 pm, David George wrote: On 2009-03-11 04:36:29 +, Mark Tolonen metolone+gm...@gmail.com said: David George d...@eatmyhat.co.uk wrote in message news:00150e67$0$27956$c3e8...@news.astraweb.com... Hi guys, I've been developing some code for a university project using Python. We've been working on an existing codebase, cleaning it up and removing dead wood. We decided to make some changes to internal message handling by using a SocketServer, which worked great when we were using 2.6, as we could simply call its shutdown() method when we wanted to stop it from 'serving forever'. Unfortunately, we've now needed to downgrade to python 2.5 to accomodate the libtorrent python bindings we need to use as part of the project. So, my question is, is there any way to stop a SocketServer that's been told to server forever in python 2.5? Sure, derive a class from one of the classes in SocketServer, and override the methods that implement the shutdown behavior in 2.6. -Mark Based on what you guys have said i've had a look at the code for serve_forever() in both 2.5 and 2.6, and tried to create my own derived class in this manner: class MBThreadingTCPServer(SocketServer.ThreadingTCPServer): def __init__(self, address_tuple, handler): SocketServer.ThreadingTCPServer.__init__(self, address_tuple, handler) self.__serving = True def serve_forever(self): while self.__serving: SocketServer.ThreadingTCPServer.handle_request(self) def shutdown(self): self.__serving = False Don't worry about the MB thing, it's just related to the name of our project. I don't think i've done this right, but i've tried to implement the serve_forever() functionality in my derived class, and also add the shutdown() method so i can stop it. Does this appear to be the right way to do things? Cheers, Dave More or less what I would do, though you should be able to call self.handle_request. It's worth noting that handle_request generally calls a blocking socket method, which means your self.__serving check only happens the next time it handles a request. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Stopping SocketServer on Python 2.5
On Mar 11, 1:11 pm, David George d...@eatmyhat.co.uk wrote: Again, problem here is the issue of being unable to kill the server while it's waiting on a request. In theory, i could force it to continue by sending some sort of junk data with the method i use to stop the server, but that seems a bit hacky, don't you think? Dave, I agree, it does. I'm in a bit over my head at this point, but does setting self.socket.settimeout(0.5) cause the call to get_request (and thus self.socket.accept()) to timeout? If so, that may be your ticket, since socket.error exceptions are already caught by the TCPServer class. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: An error in threading.py?
On Mar 11, 4:15 pm, Oltmans rolf.oltm...@gmail.com wrote: Hi, all. I'm trying to use Mechanize in a multithreaded program-- purpose of which is to fill out a form on a website using concurrent threads. Guys, trust me I've spent a lot of time to figure out the problem but I'm completed puzzled. Firstly, I've listed the errors and then the program listing (with imports omitted) Error: Traceback (most recent call last): File C:\Python25\lib\threading.py, line 460, in __bootstrap self.run() File tmechanize.py, line 21, in run with lock: NameError: global name 'lock' is not defined Time for the obvious question - where do you define lock? If you're trying to put a mutex around your increment, you need a lock pre- defined to use it. So, you'll need to add something like lock = Lock () near the top, with a global lock before you try using it in your with statement. If you try using Lock() directly in the thread, you'll create a new lock per thread, thus defeating the purpose of trying to lock in the first place. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: removing duplication from a huge list.
On Feb 27, 8:33 am, Tim Rowe digi...@gmail.com wrote: 2009/2/27 odeits ode...@gmail.com: How big of a list are we talking about? If the list is so big that the entire list cannot fit in memory at the same time this approach wont work e.g. removing duplicate lines from a very large file. We were told in the original question: more than 15 million records, and it won't all fit into memory. So your observation is pertinent. -- Tim Rowe If order did matter, and the list itself couldn't be stored in memory, I would personally do some sort of hash of each item (or something as simple as first 5 bytes, last 5 bytes and length), keeping a reference to which item the hash belongs, sort and identify duplicates in the hash, and using the reference check to see if the actual items in question match as well. Pretty brutish and slow, but it's the first algorithm which comes to mind. Of course, I'm assuming that the list items are long enough to warrant using a hash and not the values themselves. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: removing duplication from a huge list.
On Feb 27, 10:07 am, Steve Holden st...@holdenweb.com wrote: Assuming no duplicates, how does this help? You still have to verify collisions. Absolutely. But a decent hashing function (particularly since you know quite a bit about the data beforehand) will have very few collisions (theoretically no collisions, again since we know the data beforehand). The problem is you can't guarantee any hash value will be unique, so ultimately you have to check whether list entries hashing to the same value are or are not equal. Which would mean either having the whole list in memory or having access to it via some other random access method. Again, absolutely. But as Tim pointed out, with a decent hash the number of superfluous checks should be minimal, so your checks will mostly occur on valid duplicate values. It's worth noting that I picked this algorithm with the assumption that any storage to physical memory (such as required by using a set) would be out of the question. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: CSV readers and UTF-8 files
On Feb 19, 7:21 am, mk mrk...@gmail.com wrote: Hello everyone, Is it just me or CSV reader/DictReader and UTF-8 files do not work correctly in Python 2.6.1 (Windows)? I would point out in the CSV module documentation (http:// docs.python.org/library/csv.html) it explicitly mentions that it can't handle unicode. You can use their workaround in the examples section for UTF-8, or with another form of encoding (I used MIME) for UTF-16. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Will multithreading make python less popular?
On Feb 19, 1:18 pm, Paul Rubin http://phr...@nospam.invalid wrote: ... If such speedups were useless or unimportant, we would not have blown our hard earned cash replacing perfectly good older hardware, so we have to accept the concept that speed matters and ignore those platitudes that say otherwise. That's fair, but by using a high level language in the first place, you've already made the conscious decision to sacrifice speed for ease of programming. Otherwise, you would probably be programming in C. The question really is Is it fast enough, and the answer usually is Yes. And when the answer is No, there are many things which can be done before the need to multi-thread the whole script comes about. It's a proposition that used to bother me, until I did some actual programming of real world problems in Python. I've yet to really find a case where the application was slow enough to justify the cost of using multiple Python processes. ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Will multithreading make python less popular?
On Feb 19, 3:11 pm, Paul Rubin http://phr...@nospam.invalid wrote: Falcolas garri...@gmail.com writes: It's a proposition that used to bother me, until I did some actual programming of real world problems in Python. I've yet to really find a case where the application was slow enough to justify the cost of using multiple Python processes. Right, that's basically the issue here: the cost of using multiple Python processes is unnecessarily high. If that cost were lower then we could more easily use multiple cores to make oru apps faster. I was actually referring to the time cost of researching or developing a parallel execution algorithm which would be suitable for multiple processes. The system overhead of using the Python multiprocess module is fairly negligible for the systems I work on. Different languages have different trade-offs. Python's trade-offs suit us. If they don't suit you, use a language with trade-offs that do. +1 ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: urllib2 - 403 that _should_ not occur.
On Jan 11, 6:59 pm, James Mills prolo...@shortcircuit.net.au wrote: Hey all, The following fails for me: from urllib2 import urlopen f = urlopen(http://groups.google.com/group/chromium-announce/feed/rss_v2_0_msgs.xml;) For what it's worth, I've had a similar problem with the urlopen as well. Using the library default urlopen results in an error, but if I build an opener with the basic handlers, it works just fine. import urllib2 f = urllib2.urlopen(http://localhost:8000;) Traceback (most recent call last): File pyshell#1, line 1, in module f = urllib2.urlopen(http://localhost:8000;) File C:\Python25\lib\urllib2.py, line 121, in urlopen return _opener.open(url, data) File C:\Python25\lib\urllib2.py, line 380, in open response = meth(req, response) File C:\Python25\lib\urllib2.py, line 491, in http_response 'http', request, response, code, msg, hdrs) File C:\Python25\lib\urllib2.py, line 418, in error return self._call_chain(*args) File C:\Python25\lib\urllib2.py, line 353, in _call_chain result = func(*args) File C:\Python25\lib\urllib2.py, line 499, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 403: Forbidden opener = urllib2.OpenerDirector() opener.add_handler(urllib2.HTTPHandler()) opener.add_handler(urllib2.HTTPDefaultErrorHandler()) f = opener.open(http://localhost:8000;) f.read() 'something relevant' -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a better/simpler way to filter blank lines?
On Nov 4, 3:30 pm, tmallen [EMAIL PROTECTED] wrote: On Nov 4, 4:30 pm, [EMAIL PROTECTED] wrote: tmallen: I'm parsing some text files, and I want to strip blank lines in the process. Is there a simpler way to do this than what I have here? lines = filter(lambda line: len(line.strip()) 0, lines) xlines = (line for line in open(filename) if line.strip()) Bye, bearophile I must be missing something: xlines = (line for line in open(new.data) if line.strip()) xlines generator object at 0x6b648 xlines.sort() Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'generator' object has no attribute 'sort' What do you think? Thomas Using the surrounding parentheses creates a generator object, whereas using square brackets would create a list. So, if you want to run list operations on the resulting object, you'll want to use the list comprehension instead. i.e. list_o_lines = [line for line in open(filename) if line.strip()] Downside is the increased memory usage and processing time as you dump the entire file into memory, whereas if you plan to do a for line in xlines: operation, it would be faster to use the generator. -- http://mail.python.org/mailman/listinfo/python-list
Re: iterate start at second row in file not first
On May 20, 12:34 pm, [EMAIL PROTECTED] wrote:
how do i jump the first step there? i dont want to iterate the first
row...how do i start at the second?
To simply bypass the first line, do a readline() on the file object
before starting to process the file. To filter out particular lines
throughout the file, you can either do a check in the for loop and
continue if it is true, or create a generator which filters out
unwanted content.
untested:
fd = open(some_file)
fi = (x for x in fd if (x.search(':') 0))
for line in fi:
pass
It gets a bit more complicated if you want to actually split on
sentences, not lines.
--
http://mail.python.org/mailman/listinfo/python-list
Re: os.fork leaving processes behind
On Dec 28, 12:11 am, Gabriel Genellina [EMAIL PROTECTED] wrote: I'd try to use any of the existing server implementations in SocketServer.py, but if you insist on using your own, look at the ForkingMixin class as an example of using waitpid() to avoid having zombie processes. -- Gabriel Genellina Thanks for the excellent advice. I went with the SocketServer, and I'm quite happy with the results. I had not considered it earlier due to the atrocious documentation on the Python site. Google helped with that immensely. G -- http://mail.python.org/mailman/listinfo/python-list
os.fork leaving processes behind
Hi all, This may be more of a Linux question, but it relates to how Python forks... Today, I implemented a pretty simple listener script using os.fork. The script runs fine, and performs as expected, but the process table is left with an odd entry for every fork called. I'm running on Slackware 9, under the 2.4 kernel, Python 2.5.1. while True: conn, addr = s.accept() if os.fork(): continue else: handle_connection(conn) sys.exit(0) Running ps -ef results in a slew of '[ python depreciated ]' entries (or something similar, I no longer have the actual output). Will these clean themselves up if I leave the process running, and what causes these? Thanks in advance, G -- http://mail.python.org/mailman/listinfo/python-list
Functions as Objects, and persisting values
Please help me understand the mechanics of the following behavior.
def d():
header = 'I am in front of '
def e(something):
print header + something
return e
f = d()
f('this')
I am in front of this
del(d)
f('this')
I am in front of this
The way I understand it, function d is an object, as is e. However I
don't quite grok the exact relationship between e and d. Is e
considered to be a subclass of 'd', so that it has access to it's
parent's __dict__ object, in order to access the value of 'header'? Or
is this persistence managed in a different fashion?
--
http://mail.python.org/mailman/listinfo/python-list
HTML Parser for Jython
Does anybody know of a decent HTML parser for Jython? I have to do some screen scraping, and would rather use a tested module instead of rolling my own. Thanks! GP -- http://mail.python.org/mailman/listinfo/python-list
Re: HTML Parser for Jython
On Oct 17, 9:50 am, Carsten Haese [EMAIL PROTECTED] wrote: Recent releases of BeautifulSoup need Python 2.3+, so they won't work on current Jython, but BeatifulSoup 1.x will work. Thank you. -- http://mail.python.org/mailman/listinfo/python-list
Re: Copy List
On Jul 18, 6:56 am, Rustom Mody [EMAIL PROTECTED] wrote: This is shallow copy If you want deep copy then from copy import deepcopy What will a deep copy of a list give you that using the slice notation will not? -- http://mail.python.org/mailman/listinfo/python-list
Re: Python's only one way to do it philosophy isn't good?
On Jul 5, 10:30 am, Chris Mellon [EMAIL PROTECTED] wrote:
I don't think anyone has suggested that. Let me be clear about *my*
position: When you need to ensure that a file has been closed by a
certain time, you need to be explicit about it. When you don't care,
just that it will be closed soonish then relying on normal object
lifetime calls is sufficient. This is true regardless of whether
object lifetimes are handled via refcount or via true garbage
collection. Relying on the specific semantics of refcounting to give
certain lifetimes is a logic error.
For example:
f = some_file() #maybe it's the file store for a database implementation
f.write('a bunch of stuff')
del f
#insert code that assumes f is closed.
This is the sort of code that I warn against writing.
f = some_file()
with f:
f.write(a bunch of stuff)
#insert code that assumes f is closed, but correctly this time
is better.
This has raised a few questions in my mind. So, here's my newbie
question based off this.
Is this:
f = open(xyz)
f.write(wheee)
f.close()
# Assume file is closed properly.
as safe as your code:
f = some_file()
with f:
f.write(a bunch of stuff)
#insert code that assumes f is closed, but correctly this time
Thanks!
G
--
http://mail.python.org/mailman/listinfo/python-list
Re: appending file
On Jun 29, 1:04 pm, Kuo [EMAIL PROTECTED] wrote: # FPGA CLOCK^M NET SYSCLK_A loc = N16 | TNM_NET = SYSCLK_A;^M NET SYSCLK_AN loc = M16 | TNM_NET = SYSCLK_A;^M I see those bloody ^M's anytime I have to deal with a DOS file (since it's the carrage return \r character). Is 'pin' a DOS generated file? If you want to deal with a file which contains them, look for and get rid of the \r character. -- http://mail.python.org/mailman/listinfo/python-list
Capturing stdout from a class method
I have a rather strange situation, and I'm not sure my brief experience of Python will let me handle it properly. The situation is this: I have a Java class X which I need to call in a Jython script. The output of X is sent to stdout using the java call System.out. I need to capture this output, and preform tests on it in the Jython script. My first pass at a design involves two jython scripts. One (we'll call it Y) whose sole function is to instantiate and call X, and die. The second script will call Y using a method which captures stdout to a pipe. The second script would then read the stdout from the pipe and act accordingly. Can anybody suggest a better way to handle this? The Java class, X, can not be modified for this test. -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs: terminology buffer and keybinding
On Jun 22, 11:28 am, Robert Uhl [EMAIL PROTECTED] wrote: That's the advantage of a well-organised set of commands. If you want to use regexp search, you have to look at the dialogue box and click on a checkbox--which would be a context switch. Again, you are assuming that the editor isn't set up in a way which allows this to be done from the keyboard. ctrl-f, alt-e, type, enter That's even assuming that your editor _offers_ regexp search. If emacs didn't have it, I could add it, and it'd be just as much part of the editor as if it were included... One advantage I'm more than happy to cede to you - the current program I use is closed source and not extensible. Though, I'm sure that there are editors for windows/mac/xwindows which are as extensible as emacs. It's Mac OS and Windows which are inconsistent. Emacs has been around since they were mere glimmers in the eye of Jobs Gates... Inconsistent? I would have to disagree. They changed paradigms - terminal text based interfaces to GUIs. You wouldn't expect a piece of software built for a terminal to be backwards compatibility to punch card interfaces, would you? Why would a GUI based program limit itself to functionality as defined by a terminal application? -- http://mail.python.org/mailman/listinfo/python-list
Re: Tailing a log file?
On Jun 22, 12:50 pm, Kenji Noguchi [EMAIL PROTECTED] wrote: I checked the source code for tail and they actually poll the file by using fstat and sleep to check for changes in the file size. This didn't seem right so I thought about it more and realized I ought to be using inotify. So I guess I answered my own question. I built something which worked on the check stat and sleep method. Using notify or select may work, but it's not portible to Windows systems, which may or may not be an issue for you. #! /usr/bin/env python import os import sys import time from stat import * class Watchfile(object): def __init__(self, f_loc): self.file_loc = f_loc self.prev_lm = 0 self.last_read_pos = 0 f = open(self.file_loc, r) for l in f: pass self.last_read_pos = f.tell() f.close() self.prev_lm = os.stat(self.file_loc)[ST_MTIME] def changed(self): lm_time = os.stat(self.file_loc)[ST_MTIME] if lm_time self.prev_lm: return True else: return False def get_since_last_read(self): f = open(self.file_loc, r) f.seek(self.last_read_pos) lines = f.readlines() self.last_read_pos = f.tell() f.close self.prev_lm = os.stat(self.file_loc)[ST_MTIME] return lines if __name__ == __main__: x_file = Watchfile(file path) y_file = Watchfile(file path) while True: if x_file.changed(): lines = x_file.get_since_last_read() # do something if y_file.changed(): lines = y_file.get_since_last_read() # do something else try: time.sleep(5) except KeyboardInterrupt: break -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs: terminology buffer and keybinding
On Jun 22, 3:06 pm, Pascal Bourguignon [EMAIL PROTECTED] wrote: How do you call a Mac user interface that let a user work during 3 hours to do a simple modification to a MS-Word file that takes 15 seconds to do with emacs or a simple unix script? Would you mind elaborating on *what* took 3 hours to do, as opposed to just throwing around unquantified numbers? Would you also mind explaining the user's familiarity with the tools they were using on the mac? It's just as easy for me to say that it took me 30 minutes to simply exit emacs, and use that to justify that emacs, and by extension Linux, is a terrible tool. -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs: terminology buffer and keybinding
On Jun 22, 4:12 pm, Pascal Bourguignon [EMAIL PROTECTED] wrote: Anything that the user have to do repeatitively with the GUI, like copy-and-paste, or reformating of a lot of paragraphs or table entries, and which is done automatically by writting a one-liner program in emacs or shell. So the tool they were using did not support macros? You're right, that would suck. I'm guessing this is before you could expose the Unix underpinnings on the mac. I will agree that I do miss much of the standard shell utility when working in windows. Fortunately, I am able to replace a lot of that with well written python or perl scripts. And they tried to put graphical user interfaces on scripting, it doesn't work either. Programming is working with text, with verbs. Recording macros could be considered a form of programming, which can have nothing to do with any text. Granted, they're pretty dumb if you don't manually modify them, but really, nothing is stopping you from modifying them either. I can't count the number of times I've created a macro to do repeated modification of a text file. I guess ultimately I'm trying to argue the point that just because a tool was written with a GUI or on Windows does not automatically make it any less a productive tool than a text based terminal tool. Even in windows, you can use the keyboard to do all of your work, if you learn how (thanks to the magic of the alt key). -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs: terminology buffer and keybinding
On Jun 21, 10:09 am, Robert Uhl [EMAIL PROTECTED] wrote:
You're quite right. Windows/Mac user interfaces are so clunky that they
massively hamper productivity. Emacs, OTOH, enables it. For example,
C-s is search forward; C-r is search backward ('reverse'); C-M-s is
search forward for a regular expression; C-M-r is search backward for a
regular expression. A Windows or Mac editor would have C-s for save,
and that's it. It might have C-f for find, but it'd pop up a dialogue
instead of offering an interactive search, causing a mental context
switch. Searching would interrupt one's flow of thought rather than
being part of it.
Being a primarily windows user, I have to question your assertion that
using ctrl-f for find causes a mental context switch. For me, in 90%
of the windows applications, finding something is as simple as ctrl-f,
the phrase, hit enter. Not terribly different from your set of
commands. The biggest difference is that if I need to use a Find
feature I might not often use, I have a visual interface to all the
related search functions. On the other hand, a terminal program would
necessitate a memory search at best, or a trip to the help pages at
worst.
The best part of my windows knowledge is that it's transferable to
most (all?) of the applications I work with. Find is usually ctrl-f.
Undo is ctrl-z. Save is ctrl-s, yadda yadda. Such knowledge is rarely
transferable from terminal programs in my experience -- what may be
true for one program (emacs) is wildly different in another program
(vi), and useless in yet another (pico).
On the other hand, I can move from notepad to Word to Open Office to
Notepad++, based on the availability at the terminal I'm on, with
little difficulty.
For the record, I use VIM when in terminals. Emacs isn't available on
our *nix boxes.
--
http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs: terminology buffer and keybinding
On Jun 21, 2:10 pm, Kaldrenon [EMAIL PROTECTED] wrote: I don't think anyone can make the argument that any (past or current) graphics-based editor is as efficient when being used to its fullest as a text-based editor. It's basic math - it takes measurably more time to move a hand to the mouse, move/click the mouse, and more the hand back to the touch-typing position than it does to execute even a moderately complex series of keystrokes. Maybe not large amounts of time -per action-, but it doesn't take too long for it to add up if you spend a lot of time editing. Contrast the time saved by not having to reposition one's hands, the extensibility, and customization against the learning curve of an interface that doesn't exactly throw its controls at the user, and here's the conclusion I think results: graphical interfaces are - easier- to develop some proficiency with, but proficiency with emacs pays of far more in the long run. I have to point out, that this makes the assumption that the most oft- used commands in a GUI editor are not as easily accessed from the keyboard as they are in a terminal editor. I took a moment to look at the gui editor which has been made available to me, and short of the remove leading spaces commands, I do not need to remove my hands from the keyboard if I do not want to. Your statement holds true if, and only if, a user does not take full advantage of the keyboard commands. But if we're talking about experienced users in both cases, then that's not an issue, is it? -- http://mail.python.org/mailman/listinfo/python-list
