Hi Terry. I am curious which reactor are you using for this. Any threads
need to be java threads with pylucene. Not sure if this might work but
perhaps creating a thread pool with the right kind of threads could
help. Somewhere on this list a while back, someone put together some
thread pool code that you might find helpful for this. Twisted also has
its own thread pool so this would need some modification. Have you tried
testing your index outside of twisted to see whether the issue persists.
Regards,
David
Terry Jones wrote:
Hi.
I'm working on an application that using Python 2.5, PyLucene 2.1.0-2 and
Twisted trunk (revision 20059) running under Mac OS X 10.4.9.
The Twisted server is mysteriously and silently dying, in a fairly
predictable way. When I stick in debug prints to stderr, I can see that
(usually) a call to PyLucene.IndexWriter.optimize and (sometimes) a call to
PyLucene.IndexReader.open never returns.
Following a suggestion from Glyph on the twisted.web mailing list, I used
ktrace to see if any system calls were going astray. It doesn't look like
it. But the end of the ktrace (right before the process dies) reveals:
7826 Python CALL write(0x2,0xbfff971f,0x18)
7826 Python GIO fd 2 wrote 24 bytes
"thread_get_state failed
"
7826 Python RET write 24/0x18
7826 Python CALL sigprocmask(0x3,0xbfff9b08,0)
7826 Python RET sigprocmask 0
7826 Python CALL kill(0x1e92,0x6)
7826 Python RET kill 0
7826 Python PSIG SIGABRT SIG_DFL
7826 Python NAMI "/cores/core.7826"
Nothing fails before that. The "thread_get_state failed\n" message does
not appear in the twisted server log. The kill is an abort to this process
(i.e., 0x1e92 = 7826). So something (but not a system call) has gone wrong
and the code has called abort. Which explains why the server abruptly dies.
Running gdb python /cores/core.7826 provides no useful info:
(gdb) where
#0 0x00000000 in _mh_dylib_header ()
From the kdump output, the process is clearly in the middle of doing
PyLucene things (there are a bunch of access calls to files that are in my
PyLucene index directories).
Grepping for 'thread_get_state failed' in Python, Twisted, and PyLucene
gets me just one hit:
$ grep -i 'thread_get_state failed' /usr/local/lib/*
Binary file /usr/local/lib/libgcj.6.dylib matches
Which of course is a GCJ library file distributed with the Mac OS X binary
version of PyLucene.
All of which led me here. The above kdump output comes from a failed call
to PyLucene.IndexWriter.optimize. I don't _think_ I'm doing anything wrong
in my Python - the code in question gets run successfully multiple times
before it eventually falls over. I'm not using threads.
I'd be happy to get high-level feedback on this. E.g., Is this a known
issue? Should I go read the Lucene or PyLucene sources to track it down?
Would it help if I built PyLucene from scratch? Can I help to fix this? Etc.
Thanks for any help,
Terry
_______________________________________________
pylucene-dev mailing list
[email protected]
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
_______________________________________________
pylucene-dev mailing list
[email protected]
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
