Hi Python experts, I need to extend Python with C/C++ to interact with an in-house legacy API. I'm implementing a data subscriber with Python, which subscribes to a data publisher (actually a ZeroMQ publisher socket) and will get notified once any new messages are fed. In my subscriber, the messages are dumped to a data processor after received. The subscriber will also get notified by the processor when done. Since the data processor is written in C++, I have to extend the Python code with a simple C++ module.
Below is a simplified runnable code sample of my data subscriber. Code main.py , in which the module proc represents the processor, subscribe to a ZeroMQ socket on localhost:10000 , setup the callback, and send the received message to the processor by calling proc.onMsg . #!/bin/python # main.py import gevent import logging import zmq.green as zmq import pub import proc logging.basicConfig( format='[%(levelname)s] %(message)s', level=logging.DEBUG ) SUB_ADDR = 'tcp://localhost:10000' def setupMqAndReceive(): '''Setup the message queue and receive messages. ''' ctx = zmq.Context() sock = ctx.socket( zmq.SUB ) # add topics sock.setsockopt_string( zmq.SUBSCRIBE, 'Hello' ) sock.connect( SUB_ADDR ) while True: msg = sock.recv().decode( 'utf-8' ) proc.onMsg( msg ) def callback( a, b ): print( '[callback]', a, b ) def main(): '''Entrance of the module. ''' pub.start() proc.setCallback( callback ) '''A simple on-liner gevent.spawn( setupMqAndReceive ).join() works. However, the received messages will not be processed by the processor. ''' gevent.spawn( setupMqAndReceive ) proc.start() Module proc is simplified with three functions exported: * setCallback setup the callback function, so that when the message processed, my subscriber could be notified; * onMsg is invoked by the subscriber; * start setups a new worker thread to handle the messages from subscriber and make the main thread join to wait for the worker thread to exit. The full version of source code could be found on github at https://github.com/more-more-tea/python_gil . Nevertheless, it does not run as my expectation. Once the processor thread is added, the subscriber cannot receive data from the publisher in the gevent loop. If I simply drop the data processor module, the subscriber gevent loop could receive the messages from the publisher. The same behavior could be observed with threading.Thread instead of gevent. I posted the same question on StackOverflow http://stackoverflow.com/questions/37513957/does-python-gil-need-to-be-taken-care-when-work-with-multi-thread-c-extension with one answer direct me to the article https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock I read the document but still not sure where to add the PyGILState_Ensure/PyGILState_Release. Without a running sample, it is a little bit hard to understand how Python thread and the native pthread interact. Would you mind review the source code and direct the right way to make it running? Thank you very much for your time and best regards! Shuang -- https://mail.python.org/mailman/listinfo/python-list