Hello all,

as we're intensive Arrow users in different enviroments and each of them raises 
their unique set of bugs, I thought I share some insights into various things 
and give a bit background to the issues I have raised and will raise. In my 
setting, there is a Python wheel `pyarrow`, a wheel `turbodbc` that depends on 
that wheel, various pure Python packages that depend on them as well as some 
C++ code that depends on Arrow C++ (incl. libarrow_python) and Boost.

At first, we have linked Parquet & Arrow as private static libraries into the 
C++ code and used the Python wheels just as they are produced in the official 
build. This lead to the issue that parquet-cpp was linked statically in one 
case and dynamically in the other case. As parquet-cpp has global variables 
that are deconstructed on library unload / program exit, these destructors were 
crashing with a double-free in some cases. As it turns out, destructors for 
global variables are registered and called differently depending on if they 
origin from a static or a shared library. Thus if you link a library in two 
ways, they are called twice. One call succeeds, the other one crashes.

Due to this, I made the change that Arrow should now be dynamically linked 
everywhere and Parquet statically. The reason to link parquet-cpp statically 
was that there was not authoritative source for the Parquet library, i.e. we 
produce a pyarrow wheel, not a parquet wheel, so parquet should be a private 
thing of that library. Several people I asked agreed that this was the correct 
way to do such things. This works nicely until you reach the point where you 
embed the Python interpreter in a process that depends natively on parquet-cpp 
and load `pyarrow` as a Python package in there. When you end a program, all 
the destructors of all libraries are called, just fine. But when you close your 
embedded Python interpreter before the end of the whole program, all Python 
libraries are unloaded. Thus the in pyarrow statically linked parquet-cpp calls 
its destructor. Sadly at the end of the whole program, also all destructors 
libraries that were linked into the main program are called. In this case the 
parquet-cpp destructor was called a second time and we get a double free again. 
In contrast to this behaviour, if parquet-cpp was dynamically linked in both 
places, a reference counter should be held by the process and the destructor 
should only be called once on program exit: 
https://issues.apache.org/jira/browse/ARROW-2245

This is just one of the funny things we had to debug and I thought it might be 
good to share some insights on one of these packaging issues. In the end, this 
also means that we might need to rethink which packages we actually want to 
statically link in the conda-forge package. My feeling here is that especially 
the larger ones like Thrift and Boost should be dynamically linked.

My main learning from this is:
 * having control over your build toolchain is key. The pyarrow wheels will 
never be as good as the conda-forge packages. Things like 
https://issues.apache.org/jira/browse/ARROW-1975 and 
https://issues.apache.org/jira/browse/ARROW-2246
 * dynamic linking may not be the ultimate silver bullet for this problem but 

Uwe

Reply via email to