Hi, If this is circular, it's a problem. But this isn't circular for now.
I think that we can use libarrow as the fundamental shared library to provide common implementation like [1] if we need to provide common implementation for template. (I think that we don't provide common implementation for template.) [1] https://github.com/apache/arrow/pull/5221/commits/e88b2579f04451d741eeddcb6697914bcc1019a6 Anyway, I'm not strongly oppose to this idea. If we choose one shared library approach, Linux packages, GLib bindings and Ruby bindings can follow the change. Thanks, -- kou In <cajpuwmdwencjpbw+hrswaojfez7e_yci-fg2d3lwgvncf45...@mail.gmail.com> "Re: [DISCUSS][C++] Rethinking our current C++ shared library (.so / .dll) approach" on Thu, 12 Sep 2019 13:23:01 -0500, Wes McKinney <wesmck...@gmail.com> wrote: > One thing I forgot to mention: > > One of the things driving the creation of new shared libraries is > interdependencies. For example: > > libarrow -> libparquet > libarrow -> libarrow_dataset > libparquet -> libarrow_dataset > > With the modular LLVM-like approach this issue goes away. > > On Thu, Sep 12, 2019 at 1:16 PM Wes McKinney <wesmck...@gmail.com> wrote: >> >> I forgot to add the link to the LLVM library listing >> >> https://gist.github.com/wesm/d13c2844db0c19477e8ee5c95e36a0dc >> >> On Thu, Sep 12, 2019 at 1:14 PM Wes McKinney <wesmck...@gmail.com> wrote: >> > >> > hi folks, >> > >> > I wanted to share some concerns that I have about our current >> > trajectory with regards to producing shared libraries from the Arrow >> > build system. >> > >> > Currently, a comprehensive build produces many shared libraries: >> > >> > * libarrow >> > * libarrow_dataset >> > * libarrow_flight >> > * libarrow_python >> > * libgandiva >> > * libparquet >> > * libplasma >> > >> > There are some others. There are a number of problems with the current >> > approach: >> > >> > * Each DLL needs its own set of "visibility" macros to control the use >> > of __declspec(dllimport/dllexport) on Windows, which is necessary to >> > instruct the import or export of symbols between DLLs on Windows. See >> > e.g. >> > https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/visibility.h >> > >> > * Templates instantiated in one DLL may cause a violation of the One >> > Definition Rule during linking (we lost at least a day of work time >> > collectively to issues around this in ARROW-6244). It is good to be >> > able to share common template interfaces in general >> > >> > * Statically-linked dependencies in one shared lib may need to be >> > statically linked into another library. For example, libgandiva >> > statically links parts of LLVM, but we will likely have some other >> > code that makes use of LLVM for other purposes (it has been discussed >> > in the context of Avro parsing) >> > >> > Overall, my preferred solution to these issues is to move to a similar >> > approach to what the LLVM project does. To help understand, let me >> > have you first look at the libraries that come from the llvm-7-dev >> > package on Ubuntu >> > >> > Here we have a collection of static "module" libraries that implement >> > different parts of the LLVM platform. Finally, a _single_ shared >> > library libLLVM-7.so is produced. >> > >> > I think we should do the same thing in Apache Arrow. So we only ever >> > will produce a single shared library from the build. We can >> > additionally make the "name" of this shared library configurable to >> > suit different needs. For example, the default name could be simply >> > "libarrow.so" or something. But if someone wants to produce a >> > barebones Parquet shared library they can override the name to create >> > a "libparquet.so" that contains only the "libarrow_core.a" and >> > "libarrow_io.a" symbols needed for reading Parquet files. >> > >> > This would have additional benefits: >> > >> > * Use the same visibility macros for all exported C++ symbols, rather >> > than having to define DLL-specific visibility >> > >> > * Improved modularization of builds and linking for third party users, >> > similar to the way that LLVM's modular linking works, see the way that >> > Gandiva requests specific components from LLVM to use for static >> > linking >> > https://github.com/apache/arrow/blob/master/cpp/cmake_modules/FindLLVM.cmake#L53 >> > >> > * Net simpler linking and deployment. Only one shared library to deal with >> > >> > There are some drawbacks, however: >> > >> > * Our C++ Linux packaging approach would need to be changed to be more >> > LLVM-like (a single .deb/.yum package containing the C++ platform >> > rather than many packages as now) >> > >> > Interested to hear from other C++ developers. >> > >> > Thanks >> > Wes
