Karl Nelson <nelso...@llnl.gov> added the comment:
I am fairly sure this is a Python "bug" in the sense that there was some change
in undocumented change in requirements and the distutils pattern for building a
module no longer reflects that requirement. That said very likely JPype is
the only module to be affected and thus I will have to manually adjust to
account for the new requirement.
Unfortunately as far as developers, I am it so fixing it (with your assistance)
is going to have to fall on me. So lets do a run down of how this all working
so you can point me where to look.
1) JPype gets built as an ordinary setup.py CPython module. So it is possibly
a bug in the build pattern of the customizers in JPype that was exposed by
Python 3.9. I am just going to cut and paste so that it is easy to follow
along.
jpype/setup.py
```
...
from setuptools import Extension
...
import setupext
...
jpypeLib = Extension(name='_jpype', **setupext.platform.Platform(
include_dirs=[Path('native', 'common', 'include'),
Path('native', 'python', 'include'),
Path('native', 'embedded', 'include')],
sources=[Path('native', 'common', '*.cpp'),
Path('native', 'python', '*.cpp'),
Path('native', 'embedded', '*.cpp')], platform=platform,
))
```
We have two sets of customization in setup.py. Both are included from a module
called jpype/setupext/
The key one is the jpype/setupext/platform.py which defines the compiler flags.
There are two sections of interest...
jpype/setupext/platform.py contains these modifications for win32.
(So there is the Advapi32, not sure why it appears there because this section
is all before my time as this was started in 2001 and I joined in 2018)
```
static = True
if platform == 'win32':
distutils.log.info("Add windows settings")
platform_specific['libraries'] = ['Advapi32']
platform_specific['define_macros'] = [('WIN32', 1)]
if sys.version > '3':
platform_specific['extra_compile_args'] = [
'/Zi', '/EHsc', '/std:c++14']
else:
platform_specific['extra_compile_args'] = ['/Zi', '/EHsc']
platform_specific['extra_link_args'] = ['/DEBUG']
jni_md_platform = 'win32'
```
The second section is currently commented out. Though it is relevant because
JPype is exotic in one way. It is a module which is loaded in three ways.
When imported from Python it is an ordinary library (1) which will later pull
in Java which will then load library a second time (2) to bind Java native
methods. It can also be used to launch Python if Java starts the session (3).
In that case, it needs libpython.dll to be bound to module so that the Java
equivalent to LoadLibrary can work. When it does Java first manually loads
libpython.dll then loads the module and calls the hook to get Python started.
```
# This code is used to include python library in the build when starting Python
from
# within Java. It will be used in the future, but is not currently required.
# if static and sysconfig.get_config_var('BLDLIBRARY') is not None:
#
platform_specific['extra_link_args'].append(sysconfig.get_config_var('BLDLIBRARY'))
```
The actual buildext has a few minor patches so that Java libraries can run
through the normal process. But nothing seems like a good candidate
We have one section tweeking some of the build options.
```
def initialize_options(self, *args):
"""omit -Wstrict-prototypes from CFLAGS since its only valid for C
code."""
self.android = False
self.makefile = False
self.jar = False
import distutils.sysconfig
cfg_vars = distutils.sysconfig.get_config_vars()
replacement = {
'-Wstrict-prototypes': '',
'-Wimplicit-function-declaration': '',
}
tracing = self.distribution.enable_tracing
# Arguments to remove so we set debugging and optimization level
remove_args = ['-O0', '-O1', '-O2', '-O3', '-g']
for k, v in cfg_vars.items():
if not isinstance(v, str):
continue
if not k == "OPT" and not "FLAGS" in k:
continue
args = v.split()
for r in remove_args:
args = list(filter((r).__ne__, args))
cfg_vars[k] = " ".join(args)
super().initialize_options()
```
Then later we interrupt the build process for Java.
```
def build_extension(self, ext):
if ext.language == "java":
return self.build_java_ext(ext)
if self.jar:
return
print("Call build ext")
return super().build_extension(ext)
```
2) Next we have the module start. I am guessing this is not the source of the
error because adding a printf shows we never got to this point.
jpype/native/pyjp_module.cpp
```
PyMODINIT_FUNC PyInit__jpype()
{
JP_PY_TRY("PyInit__jpype");
JPContext_global = new JPContext();
// This is required for python versions prior to 3.7.
// It is called by the python initialization starting from 3.7,
// but is safe to call afterwards.
PyEval_InitThreads();
// Initialize the module (depends on python version)
PyObject* module = PyModule_Create(&moduledef);
Py_INCREF(module);
PyJPModule = module;
PyModule_AddStringConstant(module, "__version__", "1.2.1_dev0");
... initialize a bunch of resources
}
```
3) After this point we will start getting exotic. JPype plugs into Python in
with less that above board methods. In order to operate it needs to derive a
bunch of Python basic types in ways that the API does not allow. As Python does
not support true multiple inheritance of concrete types and I need to add slots
that should not exist, it will use a custom memory allocator to offset the
addresses of object pointers such that a hidden Java slot appears on types
type, object, long, double, exception, etc. To do this it will forward declare
some of the private Python symbols having to do with heap types and stacktrace
frames, etc. There is also one monkey patch for the "function" type in which
it temporarily disables the "final" field so that we create a derived class
that should not exist for Java methods, then restores the field immediately
after.
This is relevant because if the symbols it is pulling in are missing for some
reason then the dll will fail to load. However, given it will load when it
isn't compiled and not load when compiled there is not much chance the symbol
table should be different. Unless windows is really different from other
architectures, this is a long shot.
3) Okay so on to module loading
```
import _jpype
```
This statement works when the debugger is attached, when called from a .py file
but fails when called from a .pyc file. I have looked at it all the way down
to the byte code. There appears to be nothing in JPype that is going into
that process. This just jumps straight into Python core code.
4) Post load a whole lot of stuff happens. We patch in the new types, and then
use LoadLibrary to pull in jvm.dll. As we work on many JVM versions and we
won't know which is requested, JPype is not actually compiled against the jvm.
Instead it manually loads all its symbols using GetProcAddress in
jpype/native/jp_platform.cpp.
So that is all there is (dirty laundary and all). Does anything look suspect?
----------
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue42529>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
