Re: problems with statically linked Boost

[Alex Samuel]

OK.

I'll probably be able to work around this problem.  Just a couple of thoughts 
for the long term:

1. It seems mostly reasonable to treat conda as closed ecosystem as you 
describe; other C++ stuff can be deployed by other means.  However, extension 
modules are always going to have to share the Python process, so this policy 
kind of says, you can't use external C++ extension code with conda.

2. Given the policy, it seems slightly better to link Boost dynamically.  I had checked 
package metadata and shared lib dependencies, and didn't even realize it used Boost, 
until one of my colleagues actually looked at the symbol table and pointed this out.  As 
it stands, Boost is an undeclared dependency, at least "dependency" in the 
sense of pinning a version.

Thanks for your help.
Alex


On 02/17/2018 11:32 AM, Wes McKinney wrote:

It sounds like for your use case that it would be better for you to
build your own Arrow packages that use the same Boost as the rest of
your repo. You can possible use the scourge tool that Phillip built to
help with this (we're using it to build nightlies).

conda-forge is a fairly closed ecosystem under the present
circumstances -- the intent is that libraries within it are
interoperable with each other, and that packages built with
conda-forge binaries as their third party dependencies (e.g. if you
were using the boost-cpp conda-forge package) will also be able to
work. Using the conda-forge stack as an add-on to a substantial
independent C++ library stack is not (IIUC) an intended use case.

Note that there are libstdc++-related issues using conda-forge
binaries with Anaconda >= 5.0 due to the change in compilers.
Hopefully this will get fixed in the next few months.

- Wes

On Sat, Feb 17, 2018 at 11:14 AM, Alex Samuel <a...@alexsamuel.net> wrote:

OK, though if both modules linked Boost statically, I believe they would
have distinct copies of global variables.  Whether or not this causes
problems depends on whether they are purely internal or tied to external
state.  My hunch is that for Boost::regex, there wouldn't be an issue with
two complete copies in the same process, as long as they remained separate.
I still suspect our module is picking up some symbols from the copy of Boost
statically linked to Parquet rather than the one it pulls in as a shared lib
dependency.  One thing I could try is to link ours statically as well.

I wasn't aware of bcp; I'll take a look at that.

It may or may not be possible for us to build our C++ stuff against
conda-forge's Boost; I'm not sure.  We have a large C++ codebase and
distribute parts of it, particularly Python extension modules, via conda.

In general, I suspect as conda-forge grows and packages more C++ code,
issues like this are likely to become an increasing problem.  Do you know if
there is a general policy regarding how extension modules in conda packages
should link common C++ libraries like Boost?

Thanks,
Alex




On 02/17/2018 11:04 AM, Uwe L. Korn wrote:

Static linking does not really solve all problems in the Boost case as
there are global variables that are picked up across different Boost
versions. Thus if you link it statically it still has an effect on other
dependencies. In the case of boost, you can get around this by using the bcp
tool http://www.boost.org/doc/libs/1_66_0/tools/bcp/doc/html/index.html to
rename your local boost to a different namespace to avoid collisions. We
will also do thus in future with our wheel in Arrow, too. But we will not do
this for conda-packages as there the assumption is that all artefacts will
be linked against the same Boost version.

Uwe

Am 17.02.2018 um 16:55 schrieb Alex Samuel <a...@alexsamuel.net>:

Yes, we do link our internal code with a different Boost version; we
build and (conda) package it ourselves.

Why should this matter if Parquet links it statically?  If static linking
won't allow us to use our own version, why bother linking statically at all?

Thanks,
Alex

On 02/17/2018 10:52 AM, Uwe L. Korn wrote:
Hi,
The issue is here no that Boost is linked statically in one and
dynamically in another but that you link against two different boost
versions. The stacktrace shows links to Boost 1.55 whereas Arrow should be
linked against 1.65 or 1.66 (the one coming from conda-forge). Arrow
requires at least a Boost version of 1.60+ to work. The most likely guess
from my side would be that your internal modules are linked against the
system boost, not the conda-provided boost.
Uwe

Am 17.02.2018 um 16:47 schrieb Alex Samuel <a...@alexsamuel.net>:

Hi there,

Sure, I'll append top of the stack.  You can see our internal function
"asd::infra::util::start_of_date"; everything else is Boost, Python, or
libstdc++.

My understanding (though I haven't demonstrated this conclusively) is
that, because Python loads extension modules RTLD_GLOBAL, an extension
module can pick up symbols from another or its dependencies, even if the
former "usually" satisfy relocations from their own shared lib dependencies.
So, one module linking Boost statically may interfere with another that
links it dynamically, by injecting its symbols.

If necessary I can try to put together a minimal test case, but no
guarantee it will actually trigger the bug.  But it might be worth testing
my theory above first, with gdb or by some other means.

Thanks!
Alex



#0  0x00007f6ddeba4c8b in std::basic_string<char,
std::char_traits<char>, std::allocator<char>

::basic_string(std::basic_string<char, std::char_traits<char>,

std::allocator<char> > const&) ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/numexpr/../../../libstdc++.so.6

#1  0x00007f6dd4f978d6 in
boost::re_detail::cpp_regex_traits_char_layer<char>::init() ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/../../../.././libboost_regex.so.1.55.0

#2  0x00007f6dd4fdbd88 in
boost::object_cache<boost::re_detail::cpp_regex_traits_base<char>,
boost::re_detail::cpp_regex_traits_implementation<char>

::do_get(boost::re_detail::cpp_regex_traits_base<char> const&, unsigned

long) ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/../../../.././libboost_regex.so.1.55.0

#3  0x00007f6dd4fe5bb5 in boost::basic_regex<char,
boost::regex_traits<char, boost::cpp_regex_traits<char> > >::do_assign(char
const*, char const*, unsigned int) ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/../../../.././libboost_regex.so.1.55.0

#4  0x00007f6dd575a90e in
asd::infra::util::start_of_date(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, char const*) ()

     at
/prod/sys/sysasd/opt/tudor-devtools/v1.3/Linux.el6.x86_64-corei7-avx-gcc4.83-anaconda2.0.1/include/boost/regex/v4/basic_regex.hpp:382

#5  0x00007f6dd2c36734 in
boost::python::objects::caller_py_function_impl<boost::python::detail::caller<unsigned
long (*)(std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, char const*),
boost::python::default_call_policies, boost::mpl::vector3<unsigned long,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, char const*> > >::operator()(_object*, _object*) ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/util.so

#6  0x00007f6dd52ac71a in
boost::python::objects::function::call(_object*, _object*) const ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/../../../../libboost_python.so.1.55.0

#7  0x00007f6dd52aca68 in
boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous
namespace)::bind_return,
void>::invoke(boost::detail::function::function_buffer&) ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/../../../../libboost_python.so.1.55.0

#8  0x00007f6dd52b4cd3 in
boost::python::detail::exception_handler::operator()(boost::function0<void>
const&) const ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/../../../../libboost_python.so.1.55.0

#9  0x00007f6dd2c32c03 in
boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool,
boost::python::detail::translate_exception<asd::infra::Exception, void
(*)(asd::infra::Exception const&)>, boost::_bi::list3<boost::arg<1>,
boost::arg<2>, boost::_bi::value<void (*)(asd::infra::Exception const&)> >

, bool, boost::python::detail::exception_handler const&,

boost::function0<void>
const&>::invoke(boost::detail::function::function_buffer&,
boost::python::detail::exception_handler const&, boost::function0<void>
const&) ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/util.so

#10 0x00007f6dd52b4a9d in
boost::python::handle_exception_impl(boost::function0<void>) ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/../../../../libboost_python.so.1.55.0

#11 0x00007f6dd52ab2b3 in function_call ()

    from
/space/asd/conda/envs/rd-20180212-0/lib/python2.7/site-packages/asd/infra/../../../../libboost_python.so.1.55.0

#12 0x00007f6df2bc8e93 in PyObject_Call (func=0x2799850, arg=<value
optimized out>,

     kw=<value optimized out>) at Objects/abstract.c:2547

#13 0x00007f6df2c7b80d in do_call (f=<value optimized out>,
throwflag=<value optimized out>)

     at Python/ceval.c:4569

#14 call_function (f=<value optimized out>, throwflag=<value optimized
out>) at Python/ceval.c:4374

#15 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value
optimized out>) at Python/ceval.c:2989

On 02/17/2018 10:31 AM, Uwe L. Korn wrote:
Hello,
I am not sure why we are linking statically in the conda-forge
packages, as a gut feeling we should link dynamically there. Wes, can you
remember why?
Alex, would it be possible for you to send us the part of the
segmentation fault that is not private to your modules. That would be a good
indicator for us what is going wrong.
Typically it is best when you enable coredumps with `ulimit -c
unlimited` and then run your program as usual. There should be no
performance penalty. When ist segfaults, run `gdb python core` (note that
the core file might also be postfixed with the PID but that depends on your
system). In gdb type 'thread apply all bt full'. Post thd output pf that
command and strip away the parts we should not see. Most relevant will be
the stacktrace of the thread that segfaulted.
Uwe

Am 16.02.2018 um 23:17 schrieb Alex Samuel <a...@alexsamuel.net>:

Hello,

I am having some troubles using the Continuum PyArrow conda package
dependencies in conjunction with internal C++ extension modules.

Apparently, Arrow and Parquet link Boost statically.  We have some
internal packages containing C++ code that linking Boost libs dynamicaly.
If we import Feather as well as our own extension modules into the same
Python process, we get random segfaults in Boost.  I think what's happening
is that our extension modules are picking up Boost's symbols from Arrow and
Parquet already loaded into the process, rather than from our own Boost
shared libs.

Could anyone explain the policy for linking Boost in binary
distributions, particularly conda packages?  What is your expectation for
how other C++ extension modules should be built?

Thanks in advance,
Alex