Re: [Pythonmac-SIG] urllib2.urlopen fails in a forked daemon with a CoreFoundation error
On Sun, Mar 15, 2009 at 9:13 PM, Bill Janssen jans...@parc.com wrote: You need to do an exec after doing a fork. /usr/bin/python on OS X is a framework build, and some (most?) of the OS X frameworks just don't work after doing a fork. You have to restart with an exec. All right, but how is this done? I'll refer to the code I'm using: http://code.activestate.com/recipes/278731/ The first fork happens at line 55, and the second at line 103. As said, I really don't understand this fork/exec business so I don't know what I should do. Which of the many exec* functions in the os module should I run, and with what parameters? The forking without exec already does everything I need it to do daemonizing-wise (except satisfy CoreFoundation), so what am I trying to accomplish? What does CF actually require me to do, and does it mess up or require further changes in the forking methodology I'm using? - Jarkko L. ___ Pythonmac-SIG maillist - Pythonmac-SIG@python.org http://mail.python.org/mailman/listinfo/pythonmac-sig
Re: [Pythonmac-SIG] urllib2.urlopen fails in a forked daemon with a CoreFoundation error
Jarkko Laiho jarkko.la...@iki.fi wrote: So ultimately the question is: how should the forking code in the recipe be modified, so that it satisfies the requirements of OS X? What does an actual, concrete exec() call look like in this case, and at what point in the code should it occur? I don't know -- you didn't post your entire program. Read the Wikipedia page and follow the refs. Here's an example: def daemon (): try: pid = os.fork() if pid 0: sys.exit(0) # parent except OSError, e: msg = fork #1 failed: (%d) %s\n % (e.errno, e.strerror) sys.stderr.write(msg) sys.exit(1) os.umask(0) # now create a new session os.setsid() # and fork into the new session try: pid = os.fork() if pid 0: sys.exit(0) # session leader exits here except OSError, e: msg = fork #2 failed: (%d) %s\n % (e.errno, e.strerror) sys.stderr.write(msg) sys.exit(1) # here you could make any changes you need to the env, like setting PYTHONPATH env = os.environ.copy() # figure out how to invoke the actual program args = [sys.executable, -c, import mycode; + mycode.realmain() ] os.execve(sys.executable, args, env) def realmain(): start_server() # the real main... # this is basically just an event-handling loop while True: try: asyncore.loop() except (KeyboardInterrupt, SystemExit), x: note(4, Exited from main loop due to exception:\n%s, ''.join(traceback.format_exception(*sys.exc_info( raise except: note(0, Exited from main loop due to exception:\n%s, ''.join(traceback.format_exception(*sys.exc_info( if __name__ == __main__ and (not sys.platform.lower().startswith(win)): daemon() ___ Pythonmac-SIG maillist - Pythonmac-SIG@python.org http://mail.python.org/mailman/listinfo/pythonmac-sig
Re: [Pythonmac-SIG] urllib2.urlopen fails in a forked daemon with a CoreFoundation error
What does an actual, concrete exec() call look like in this case, and at what point in the code should it occur? I don't know -- you didn't post your entire program. Read the Wikipedia page and follow the refs. I did and understood just about none of it. Way over my head. But: Here's an example: [...] # figure out how to invoke the actual program args = [sys.executable, -c, import mycode; + mycode.realmain() ] os.execve(sys.executable, args, env) [...] There's a technique I didn't know about. I didn't think there was any way of making an exec() call fit the existing code, but that might just do the trick. I'm pretty sure I can work with that. Thanks for your patient advice! - Jarkko L. ___ Pythonmac-SIG maillist - Pythonmac-SIG@python.org http://mail.python.org/mailman/listinfo/pythonmac-sig
Re: [Pythonmac-SIG] urllib2.urlopen fails in a forked daemon with a CoreFoundation error
You need to do an exec after doing a fork. /usr/bin/python on OS X is a framework build, and some (most?) of the OS X frameworks just don't work after doing a fork. You have to restart with an exec. It's hard to say just what is using CF; I see you're importing some external packages (json doesn't come with 2.5.1 on OS X), so it might be one of them, instead of urllib2. But it's probably the urllib module, which uses CF to call into the SystemConfiguration system on the Mac to look up proxies and such. It would make sense that the error is triggered just when you call urlopen(). Bill ___ Pythonmac-SIG maillist - Pythonmac-SIG@python.org http://mail.python.org/mailman/listinfo/pythonmac-sig
[Pythonmac-SIG] urllib2.urlopen fails in a forked daemon with a CoreFoundation error
Hi all,
I'm writing a simple daemon to test out a web application that
receives messages via HTTP, using urllib2 for the HTTP connections.
The program daemonizes nicely and sends the messages just fine, but
when it's sending the first message (which takes place 2-12 seconds
after starting the program, since the send interval is randomized),
the terminal I started the program from prints out:
The process has forked and you cannot use this CoreFoundation
functionality safely. You MUST exec().
Break on
__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__()
to debug.
...repeating it several dozen times.
The execution of the program seems to run normally regardless, at
least the messages arrive to the HTTP server just fine.
I'm running OS X Leopard 10.5.6, the system default Python 2.5.1, and
using this daemonizing code:
http://code.activestate.com/recipes/278731/
The actual offending bit in my own code is inside __main__, and occurs
in the main loop of the program, right after the daemonization has
taken place:
[...code...]
while True:
time.sleep(int(random.random() * 10 + 2))
params = urllib.urlencode({'json':
json.dumps({'msg': random_items(phrases)[0][1],
'chan': '07101237-20ad-4229-b663-0c3dd46b8706'}
)
})
req = urllib2.Request('http://localhost:8000/json/new_msg/',
params, headers)
try:
fd = urllib2.urlopen(req) # THIS PRODUCES THE ERROR
response = fd.read()
except urllib2.HTTPError, e:
sys.exit(1)
[...more code...]
I commented and gradually uncommented bunches of code until the error
disappeared, and I narrowed it down to the urllib2.urlopen call: if it
(and the next row that depends on it) are commented out, the error
doesn't appear.
Googling the error revealed this Leopard release notes document from Apple:
http://developer.apple.com/releasenotes/CoreFoundation/CoreFoundation.html
Relevant quote:
CoreFoundation and fork()
Due to the behavior of fork(), CoreFoundation cannot be used on the
child-side of fork(). If you fork(), you must follow that with an
exec*() call of some sort, and you should not use CoreFoundation APIs
within the child, before the exec*(). The applies to all higher-level
APIs which use CoreFoundation, and since you cannot know what those
higher-level APIs are doing, and whether they are using CoreFoundation
APIs, you should not use any higher-level APIs either. This includes
use of the daemon() function.
Additionally, per POSIX, only async-cancel-safe functions are safe to
use on the child side of fork(), so even use of lower-level
libSystem/BSD/UNIX APIs should be kept to a minimum, and ideally to
only async-cancel-safe functions.
This has always been true, and there have been notes made of this on
various Cocoa developer mailling lists in the past. But CoreFoundation
is taking some stronger measures now to enforce this limitation, so
we thought it would be worthwhile to add a release note to call this
out as well. A message is written to stderr when something uses API
which is definitely known not to be safe in CoreFoundation after
fork(). If file descriptor 2 has been closed, however, you will get no
message or notice, which is too bad. We tried to make processes
terminate in a very recognizable way, and did for a while and that was
very handy, but backwards binary compatibility prevented us from doing
so.
Now, I know nothing about CoreFoundation and just about nothing about
the Unix side of things (fork/exec etc.), but it appears that
urllib2.urlopen is somehow dependent on some CoreFoundation code and
thus doesn't play well with fork(). If one of the os.exec* functions
need to be used, like the release notes suggest, I have no idea how
and where.
Is this a bug that should be reported either to Apple or to the Python
people, or do I simply need a different type of forking semantic in a
Leopard system?
Any advice will be much appreciated.
- Jarkko Laiho
___
Pythonmac-SIG maillist - Pythonmac-SIG@python.org
http://mail.python.org/mailman/listinfo/pythonmac-sig
