Hi Python experts, 

I need to extend Python with C/C++ to interact with an in-house legacy API. I'm 
implementing a data subscriber with Python, which subscribes to a data 
publisher (actually a ZeroMQ publisher socket) and will get notified once any 
new messages are fed. In my subscriber, the messages are dumped to a data 
processor after received. The subscriber will also get notified by the 
processor when done. Since the data processor is written in C++, I have to 
extend the Python code with a simple C++ module. 



Below is a simplified runnable code sample of my data subscriber. Code main.py 
, in which the module proc represents the processor, subscribe to a ZeroMQ 
socket on localhost:10000 , setup the callback, and send the received message 
to the processor by calling proc.onMsg . 


#!/bin/python
# main.py

import gevent
import logging
import zmq.green as zmq

import pub 
import proc

logging.basicConfig( format='[%(levelname)s] %(message)s', level=logging.DEBUG )

SUB_ADDR = 'tcp://localhost:10000'

def setupMqAndReceive():
    '''Setup the message queue and receive messages.
    '''
    ctx  = zmq.Context()
    sock = ctx.socket( zmq.SUB )
    # add topics
    sock.setsockopt_string( zmq.SUBSCRIBE, 'Hello' )

    sock.connect( SUB_ADDR )

    while True:
        msg = sock.recv().decode( 'utf-8' )
        proc.onMsg( msg )

def callback( a, b ):
    print( '[callback]',  a, b ) 

def main():
    '''Entrance of the module.
    '''
    pub.start()
    proc.setCallback( callback )
    '''A simple on-liner
    gevent.spawn( setupMqAndReceive ).join()
    works. However, the received messages will not be
    processed by the processor.
    '''
    gevent.spawn( setupMqAndReceive )
    proc.start() 


Module proc is simplified with three functions exported: 

    * setCallback setup the callback function, so that when the message 
processed, my subscriber could be notified; 
    * onMsg is invoked by the subscriber; 
    * start setups a new worker thread to handle the messages from subscriber 
and make the main thread join to wait for the worker thread to exit. 


The full version of source code could be found on github at 
https://github.com/more-more-tea/python_gil . Nevertheless, it does not run as 
my expectation. Once the processor thread is added, the subscriber cannot 
receive data from the publisher in the gevent loop. If I simply drop the data 
processor module, the subscriber gevent loop could receive the messages from 
the publisher. The same behavior could be observed with threading.Thread 
instead of gevent. 




I posted the same question on StackOverflow 

http://stackoverflow.com/questions/37513957/does-python-gil-need-to-be-taken-care-when-work-with-multi-thread-c-extension
 

with one answer direct me to the article 

https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock
 

I read the document but still not sure where to add the 

PyGILState_Ensure/PyGILState_Release. Without a running sample, it is a little 
bit hard to understand how Python thread and the native pthread interact. 

Would you mind review the source code and direct the right way to make it 
running? 

Thank you very much for your time and best regards! 
Shuang 
-- 
https://mail.python.org/mailman/listinfo/python-list

Reply via email to