Dear Arrow maintainers, I work on several TFX (TensorFlow eXtended) [1] projects (e.g. TensorFlow Data Validation [2]) and am trying to use Arrow in them. These projects are mostly written in Python but has C++ code as Python extension modules, therefore we use both Arrow’s C++ and Python APIs. Our projects are distributed through PyPI as binary packages.
The python extension modules are compiled with the headers shipped within pyarrow PyPI binary package and are linked with libarrow.so and libarrow_python.so in the same package. So far we’ve seen two major problems: * There are STL container definitions in public headers. It causes problems because the binary for template classes is generated at compilation time. And the definition of those template classes might differ from compiler to compiler. This might happen even if we use a different GCC version than the one that compiled pyarrow (for example, the layout of std::unordered_map<> has changed in GCC 5.2 [3], and arrow::Schema used to contain an std::unordered_map<> member [4].) One might argue that everyone releasing manylinux1 packages should use exactly the same compiler, as provided by the pypa docker image, however the standard only specifies the maximum versions of corresponding fundamental libraries [5]. Newer GCC versions could be backported to work with older libraries [6]. A recent change in Arrow [7] has removed most (but not all [8]) of the STL members in publicly accessible class declarations and will resolve our immediate problem, but I wonder if there is, or there should be an explicit policy on the ABI compatibility, especially regarding the usage of template functions / classes in public interfaces? * Our wheel cannot pass “auditwheel repair” I don’t think it’s correct to pull libarrow.so and libarrow_python.so into our wheel and have user’s Python load both our libarrow.so and pyarrow’s, but that’s what “auditwheel repair” attempts to do. But if we don’t allow auditwheel to do so, it refuses to stamp on our wheel because it has “external” dependencies. This seems not an Arrow problem, but I wonder if others in the community have had to deal with similar issues and what the resolution is. Our current workaround is to manually stamp the wheel. Thanks, Zhuo References: [1] https://github.com/tensorflow/tfx [2] https://github.com/tensorflow/data-validation [3] https://github.com/gcc-mirror/gcc/commit/54b755d349d17bb197511529746cd7cf8ea761c1#diff-f82d3b9fa19961eed132b10c9a73903e [4] https://github.com/apache/arrow/blob/b22848952f09d6f9487feaff80ee358ca41b1562/cpp/src/arrow/type.h#L532 [5] https://www.python.org/dev/peps/pep-0513/#id40 [6] https://github.com/pypa/auditwheel/issues/125#issuecomment-438513357 [7] https://github.com/apache/arrow/commit/7a5562174cffb21b16f990f64d114c1a94a30556 [8] https://github.com/apache/arrow/blob/a0e1fbb9ef51d05a3f28e221cf8c5d4031a50c93/cpp/src/arrow/ipc/dictionary.h#L91
