On Wed, 22 Apr 2020 08:44:37 +0200 Davide Andreoli <[email protected]>
said:
> I tried to link python-efl modules to python3.so and it indeed solve the
> issue for python-efl itself,
> but the problem just moved down the chain as now I get the same error in
> other python libs used by the gadgets, es:
>
> > Traceback (most recent call last):
> > File "/usr/local/lib/enlightenment/edgar_gadgets/dropbox/__init__.py",
> > line 5, in <module>
> > import socket
> > File "/usr/lib/python3.8/socket.py", line 49, in <module>
> > import _socket
> > ImportError: /usr/lib/python3.8/lib-dynload/_
> > socket.cpython-38-x86_64-linux-gnu.so: undefined symbol:
> > PyUnicode_FromFormat
> >
> this is the dropbox gadgets that import the socket module and the socket
> module (as all the py modules) is not directly linked
> to python... we are back to the starting problem :(
>
> At this point the only solution i see is to put
>
> > dlopen("libpython3.so", (RTLD_NOW | RTLD_GLOBAL));
yup. that still is good as it allows you to choose pu2 or py3 at runtime now :)
> in the edgar module code, knowing that edgar cannot coexist with other e
> modules that will try to embed a different version
> of python... well, tbh, I'm quite sure this will never happen ;)
indeed it is a theoretical problem, but it actually will be practical in som
other cases like imagine someone decides to use protobuf libs talk to some
other service and protobuf does come from google and google love to break abi
often. imagine 2 modules linking to or dlopening in global 2 incompatible
protofub libs... :)
> Any other ideas?
not really... :(
> Il giorno lun 20 apr 2020 alle ore 23:31 Carsten Haitzler <
> [email protected]> ha scritto:
>
> > On Mon, 20 Apr 2020 20:29:05 +0200 Davide Andreoli <[email protected]
> > >
> > said:
> >
> > > your first idea was in the right direction !
> > >
> > > if I change e_module_new to:
> > > dlopen(modpath, (RTLD_NOW | RTLD_GLOBAL));
> > > edgar works again as expected \o/
> >
> > ooh i just removed local.. didn't explicitly add global
> >
> > > I think this make sense because the edgar module.so is linked to
> > python.so,
> > > but the python-efl bindings modules are not. Thus importing python
> > modules
> > > require that the caller (the python interpreter or any app embedding
> > > python)
> > > must be linked to python.so. At the end RTLD_LOCAL make the linked
> > > python.so not "reachable" from the python-efl modules.
> > >
> > > Are there some reasons to not use RTLD_GLOBAL in e_module_new?
> >
> > i spotted lots of modules with the same symbols. these get loaded into the
> > global namespace of e and there is now only pure luck keeping things
> > working as
> > now we have multiple symbols with the same name and different memory
> > locations.
> > it was bad engineering to expose all those symbols. people will start to
> > create
> > symbols in one module then dlopen them or just call them in another without
> > explicit dependency handling. there has to be namespacing... and one
> > problem is
> > all modules expose the SAME public symbols because it's their public api.
> > we
> > dlsym these from the module handle but they get loaded into a global table
> > still. it's just luck that nothing broke.
> >
> > now one big reason is dependencies. for example lbpython2.7 vs libpython3.8
> >
> > let's imagine we have 2 modules that use python as a scripting engine. 2
> > edgar
> > like things. one links to python2.x and one to python3.x
> >
> > these libs break api ... BUT they share symbols they export. here's a list
> > of
> > some symbols that are the same:
> >
> > _Py_add_one_to_index_C
> > _Py_add_one_to_index_F
> > Py_AddPendingCall
> > _Py_alias
> > PyArena_AddPyObject
> > PyArena_Free
> > PyArena_Malloc
> > PyArena_New
> > _PyArg_NoKeywords
> > PyArg_Parse
> > _PyArg_Parse_SizeT
> > PyArg_ParseTuple
> > PyArg_ParseTupleAndKeywords
> > _PyArg_ParseTupleAndKeywords_SizeT
> > _PyArg_ParseTuple_SizeT
> > _Py_arguments
> > PyArg_UnpackTuple
> > PyArg_VaParse
> > ... there are over 780 of these symbols that conflict.
> >
> > so by e dlopening a module with GLOBAL instead of LOCAL... the libs the
> > module
> > links to automatically fill global symbol namespace with all of these
> > symbols... if you have 2 modules that get these syms from different .so's -
> > especially abi incompatible so's like libpython... then it's a matter of
> > luck
> > which symbol gets found first when a binding is loaded. that's why i moved
> > to
> > LOCAL... to stop these accidents. this can happen with 2 libjpegs that may
> > be
> > abi incompatible or 2 libgifs or libpngs or any library at all whre 2
> > non-identical versions may exist and different modules may implicitly bring
> > them in as a linked dep...
> >
> > so this is why i might say it might be best to not link libpython and
> > explicitly dlopen it and in this case be explicit about exposing symbols,
> > or
> > find some way of getting the python bindings and the python interpreter
> > lib to
> > bind together within a local namespace. e.g. dlopen or link to the right
> > libpython interpreter and have the python bindings link also to the right
> > python
> > interpreter lib. then the bindings will bring in their own deps and find
> > the
> > right symbols in the right lib... and it'll be in local namespace for that
> > module.
> >
> > > Il giorno ven 17 apr 2020 alle ore 23:56 Carsten Haitzler <
> > > [email protected]> ha scritto:
> > >
> > > > On Fri, 17 Apr 2020 23:20:25 +0200 Davide Andreoli <
> > [email protected]
> > > > >
> > > > said:
> > > >
> > > > > I found a way to get a bit more info, the python error while
> > importing
> > > > and
> > > > > failing is:
> > > > > ImportError: /usr/lib/python3.8/site-packages/efl/
> > > > > eo.cpython-38-x86_64-linux-gnu.so: undefined symbol:
> > > > PyUnicode_FromFormat
> > > >
> > > > oooooh. ... so it is a symbol problem. :) That i believe is provided by
> > > > libpython3.8.so ... edgar is linked to this... my guess is then
> > python is
> > > > dlopening the pytno-efl binding .so files and expecting the symbols to
> > link
> > > > up... but they don't.
> > > >
> > > > an idea. dlopen libpython3.8.so and use RTDL_GLOBAL to put python
> > symbols
> > > > in
> > > > the global namespace instead of linking to libpython at compile time -
> > do
> > > > it
> > > > runtime? then edgar can also choose at runtime either python 2 or
> > python3
> > > > libs
> > > > - dlsym the symbols you need from these... ? just trying the dlopen and
> > > > stuffing it into global namespace may help...
> > > >
> > > > > eo.cpython-38-x86_64-linux-gnu.so is the python-efl module that bind
> > > > the eo
> > > > > library,
> > > > > PyUnicode_FromFormat is a normal CPython function that should work...
> > > > > the edgar module does not get correctly linked to python?
> > > > >
> > > > > Il giorno ven 17 apr 2020 alle ore 21:54 Carsten Haitzler <
> > > > > [email protected]> ha scritto:
> > > > >
> > > > > > On Fri, 17 Apr 2020 19:33:30 +0200 Davide Andreoli <
> > > > [email protected]
> > > > > > >
> > > > > > said:
> > > > > >
> > > > > > hmm no. removing RTLD_LOCAL makes no difference. so there goes that
> > > > idea. i
> > > > > > have python-efl bindings installed - i can run ecoonman-bin as a
> > test.
> > > > > > there is
> > > > > > no output i see from python complaining... knowing what exactly
> > fails
> > > > will
> > > > > > be
> > > > > > key to finding out why. like cant find a symbol? missing
> > > > link/dependency on
> > > > > > load? something else? as there is nothing telling me what... it
> > needs
> > > > much
> > > > > > deeper digging
> > > > > >
> > > > > > > Hi guys,
> > > > > > > I'm lost in trying to fix the Edgar module to run again in E.
> > Since
> > > > some
> > > > > > > months the python import machinery does not work anymore and the
> > > > edgar
> > > > > > > module is not able anymore to import the python efl bindings.
> > > > > > >
> > > > > > > Nothing has changed in Edgar nor in python-efl and I'm able to
> > > > correctly
> > > > > > > use the machinery in a simple test app (attached), it just does
> > not
> > > > work
> > > > > > > when used inside an E module. I suspect something related to the
> > new
> > > > > > meson
> > > > > > > build, but not sure at all.
> > > > > > >
> > > > > > > The simple test attached can be built as:
> > > > > > > gcc -o test_py test_py.c `python-config --cflags --libs --embed`
> > > > > > > `pkg-config --libs --cflags efl-ui`
> > > > > > > when run it print OK, that means the python-efl module has been
> > > > found and
> > > > > > > linked. While the same code in edgar fails on the
> > import_efl__eo()
> > > > call.
> > > > > > > Note the test also need the attached .h file to build
> > > > > > >
> > > > > > > To build/run this simple code of course python-efl must be
> > installed
> > > > > > >
> > https://phab.enlightenment.org/w/projects/python_bindings_for_efl/
> > > > > > >
> > > > > > > The edgar module instead is documented here:
> > > > > > > https://phab.enlightenment.org/w/emodules/edgar/
> > > > > > >
> > > > > > > Any help/ideas would be appreciated as I really don't have any
> > idea
> > > > atm
> > > > > > >
> > > > > > > The same issue has been recently raised also on the e-user
> > mailing
> > > > list
> > > > > > >
> > > > > > > Thanks
> > > > > > > DaveMDS
> > > > > >
> > > > > >
> > > > > > --
> > > > > > ------------- Codito, ergo sum - "I code, therefore I am"
> > > > --------------
> > > > > > Carsten Haitzler - [email protected]
> > > > > >
> > > > > >
> > > >
> > > >
> > > > --
> > > > ------------- Codito, ergo sum - "I code, therefore I am"
> > --------------
> > > > Carsten Haitzler - [email protected]
> > > >
> > > >
> >
> >
> > --
> > ------------- Codito, ergo sum - "I code, therefore I am" --------------
> > Carsten Haitzler - [email protected]
> >
> >
--
------------- Codito, ergo sum - "I code, therefore I am" --------------
Carsten Haitzler - [email protected]
_______________________________________________
enlightenment-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
