Ricardo Dias schrieb am 10.12.18 um 14:42: > In the recent Cython 0.29 version was introduced a commit [1] that > hinders the usage of python subinterpreters. > > I discovered this the hard way when suddenly a component I was working > on started to crash. The component in question is the ceph-mgr daemon > from the Ceph project [2]. > > Python subinterpreters are the basic building block for the > plugin/module architecture of ceph-mgr. Each "manager module" runs in > its own python subinterpreter. Furthermore, all python bindings for the > client libraries of Ceph, such as librados, librbd, libcephfs, and > librgw, are implemented as Cython modules, and in the particular case of > librados, all ceph-mgr plugin modules import the rados Cython module > upon initialization. > > In practice, with Cython 0.29 we can only load one module, because the > following modules will refuse to load. > > After discovering this issue, we "temporarily" prevent the issue by > restricting the version of Cython as a dependency [3]. But we don't want > to keep this restriction indefinitely and would prefer a fix from the > Cython side. > > Do you think it's feasible to implement a flag to disable the safe guard > introduced in [1]? That way we could re-enable subinterpreters at our > own risk. > > [1] > https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50 > [2] https://github.com/ceph/ceph > [3] https://github.com/ceph/ceph/pull/25328
My guess is that your modules just silently leaked object references and memory with the previous Cython versions. That is why we now inserted a guard that detects cases where the module init function is executed multiple times, which would overwrite the state of the previous run. The shared library of an extension module is only loaded once, so any global C state is shared for the entire process, regardless of how often CPython calls the module init function. I am surprised that your setup didn't crash in any way. Could you explain a bit more how you are using this feature? Are the different subinterpreters running in parallel or sequentially? The ceph repo looks huge. Any pointers where I should start looking? I actually wonder if we could at least support sequential usages through the module cleanup mechanism. Once a module is cleaned up and all global objects freed, calling the module init function again should be ok. Apart from that, here is the feature ticket for module specific global state: https://github.com/cython/cython/issues/2343 Stefan _______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
