Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
I think it's possible: I chucked away my test container, but I vaguely recall it was in something in the sqlite ogr driver labelled "test"? There's a couple defined at the end of ogr/ogrsf_frmts/sqlite/CMakeLists.txt — it's possible commenting those blocks out will make the compile complete with [core,tools,sqlite3]. There may be some downsides: I don't know off the top of my head how the GDAL SQLite/GPKG integration works wrt exposing the necessary GDAL functions into SQLite. It might just work though :-) test_load_virtual_ogr and my_test_sqlite3_ext in ogr/ogrsf_frmts/sqlite/CMakeLists.txt can be safely disabled. Patch welcome to build them only under a "if (BUILD_SHARED_LIBS)", if that may help. Although I don't see why they wouldn't build with static linking. That would mean that SQLite::SQLite3 doesn't fully capture its dependencies for static builds, which may be well possible -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
Hi, On Wed, 21 Feb 2024 at 11:18, Michael Otto wrote: > > I've also tried the problem with libtiff, but couldn't solve it. Thanks > for the tip! With "gdal[core,tools]" the apps are now static. Excellent. > BUT I want to use exactly what is still causing problems: SQLite for the > creation of MBTiles (possibly GPKG) . Inevitably! > Is there perhaps still a possibility to use SQLite3 in the static apps if > I continue my first attempt without vcpkg? I think it's possible: I chucked away my test container, but I vaguely recall it was in something in the sqlite ogr driver labelled "test"? There's a couple defined at the end of ogr/ogrsf_frmts/sqlite/CMakeLists.txt — it's possible commenting those blocks out will make the compile complete with [core,tools,sqlite3]. There may be some downsides: I don't know off the top of my head how the GDAL SQLite/GPKG integration works wrt exposing the necessary GDAL functions into SQLite. It might just work though :-) You'll need to: * copy the vcpkg/ports/gdal/ dir to custom-ports/gdal/ * add a patch with the change to the sqlite CMakeLists.txt file * add your patch name to the top of custom-ports/gdal/portfile.cmake You *probably* want to add [geos,png] as well. That might throw up some more fun. Rob :) > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
Hi Simon, On Tue, 20 Feb 2024 at 18:58, Simon Eves via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > We still have one VERY strange issue whereby FlatGeoBuf export fails in a > very consistent and reproducible form down in the flatbuffer code, but only > in the static build, and only in the full system. I have written a simple > test harness that links the very same static libgdal and does a simple GDAL > startup and FGB export of a single feature and that works fine. It's some > kind of data/stack corruption when it first tries to write to the > flatbuffer on the first feature, which results in a pointer member of the > buffer class becoming 0x1000 (always) instead of null, and then it > stops on an assert. There is also one private function in the > vector_downward class which the debugger won't even step into in that > build. I can even put printfs in that function and they don't come out. > I've tried it on CentOS and on Ubuntu, with GCC and Clang, and it's always > the same. Everything else in GDAL works just fine (we have LOTS of > import/export unit tests). This makes zero sense as all the FGB code is > internal to GDAL and compiled together. I've been poking at it for over a > week and it's doing my head in. > One cause of this sort of crash is a header/library mismatch somewhere where a function is expecting different parameters/types than the caller is actually providing. Otherwise, maybe a bug in glibc/libstdc++/gcc/something that's been fixed in the intervening ten years since CentOS 7 was released? If you run your *build* on a modern distro/libc/gcc/etc does it change things? If it's the same, maybe hints more towards the former. ASAN (https://github.com/google/sanitizers/wiki/AddressSanitizer) might help track down stack/heap corruption. Rob :) ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
One of our build variants is a fully-static (or as-static-as-possible) done on CentOS 7 for old-Linux compatibility. In reverse-dependency order we have static builds of SQLite 3450100 Expat 2.5.0 KML 1.2.0 HDF5 1.12.1 NetCDF 4.8.1 TIFF 4.5.1 Proj 9.3.0 GeoTIFF 1.7.1 LittleCMS 2.15 WebP 1.3.2 Zstd 1.4.8 OpenJPEG 2.5.0 GDAL 3.7.3 We use GEOS as well, but directly, not linked into GDAL. We do our own builds of TIFF and GeoTIFF since they are also dependencies of other libs we use (e.g. pdal for LIDAR import) and the rest are to activate desirable GDAL drivers. LittleCMS, WebP, Zstd, and OpenJPEG were added recently in order to activate the GDAL JPEG2000 driver (for Sentinel2 satellite raster data) and it took some time to get all the options right so the static stack would work. I can provide the build options we use for the static, if that would be helpful. I never got the GDAL tools compiling as full-static, but that's not important for us, as we don't bundle those with our product. Our development systems are Ubuntu with more conventional dynamic builds. We still have one VERY strange issue whereby FlatGeoBuf export fails in a very consistent and reproducible form down in the flatbuffer code, but only in the static build, and only in the full system. I have written a simple test harness that links the very same static libgdal and does a simple GDAL startup and FGB export of a single feature and that works fine. It's some kind of data/stack corruption when it first tries to write to the flatbuffer on the first feature, which results in a pointer member of the buffer class becoming 0x1000 (always) instead of null, and then it stops on an assert. There is also one private function in the vector_downward class which the debugger won't even step into in that build. I can even put printfs in that function and they don't come out. I've tried it on CentOS and on Ubuntu, with GCC and Clang, and it's always the same. Everything else in GDAL works just fine (we have LOTS of import/export unit tests). This makes zero sense as all the FGB code is internal to GDAL and compiled together. I've been poking at it for over a week and it's doing my head in. Not meaning to derail this thread, but just wondering if anyone else had encountered any strange run-time behavior with static builds. On Mon, Feb 12, 2024 at 4:02 AM Michael Otto via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hello, > > I have been trying for some time to compile GDAL and the GDAL apps as a > static library / executable programs. > > The goal is to cast GDAL and all its dependencies (PROJ / GEOS / all > dependencies to system libraries / ...) into a static library and to create > the GDAL apps as static executable programs. > There should be no dynamic dependencies. > > Unfortunately, I have not had any success so far. The library is created > statically but the apps are not yet. > Does anyone have experience with this topic or possibly a procedure that > leads to success? > > Best regards. > Michael Otto > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
Hi,
On Tue, 20 Feb 2024 at 12:50, Michael Otto
wrote:
>
> I have now tried it with 'vcpkg install gdal[tools]'. The tools/apps are
> there, but they are not statically linked!
>
> See for example gdalinfo:
>
> root@vmuser-VirtualBox:/home/vmuser/Git/vcpkg/installed/x64-linux/tools/gdal#
> ldd gdalinfo
> linux-vdso.so.1 (0x7ffcc0fb3000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7fe3a31ae000)
> libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
> (0x7fe3a2f82000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7fe3a2d59000)
> /lib64/ld-linux-x86-64.so.2 (0x7fe3a6b09000)
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x7fe3a2d39000)
You're closer! GDAL itself is statically linked into those executables, as
are its dependencies...
So, I did a test, and I think this is what you need:
$ mkdir custom-triplets
$ cp vcpkg/triplets/x64-linux.cmake custom-triplets/x64-linux-static.cmake
(a "triplet" is a vcpkg configuration for a particular system/cpu/build
configuration — linux vs windows vs android, static vs dynamic, x64 vs arm,
etc)
Then edit custom-triplets/x64-linux-static.cmake and change the following
to tell it to statically link the system libraries as well:
- set(VCPKG_CRT_LINKAGE dynamic)
+ set(VCPKG_CRT_LINKAGE static)
Then build GDAL using your new triplet:
$ vcpkg/vcpkg --overlay-triplets=custom-triplets --triplet=x64-linux-static
install gdal[tools]
But of course it's not quite that simple... libtiff needs a fix too:
$ mkdir custom-ports
$ cp -a vcpkg/ports/tiff custom-ports/tiff
Then edit custom-ports/tiff/FindCMath.patch and change the following line:
- +find_library(CMath_LIBRARY NAMES m PATHS
${CMAKE_C_IMPLICIT_LINK_DIRECTORIES})
+ +find_library(CMath_LIBRARY NAMES libm.a m PATHS
${CMAKE_C_IMPLICIT_LINK_DIRECTORIES})
(this is super-hacky, there will definitely be a cleaner fix than this, but
it works for this purpose right now)
And there's a static linking issue with something SQLite-plugin-related in
GDAL. So lets reduce GDAL to just the core dependencies and the tools:
$ vcpkg/vcpkg --overlay-triplets=custom-triplets --triplet=x64-linux-static
--overlay-ports=custom-ports install gdal[core,tools]
... go and play in the sunshine ...
$ vcpkg/installed/x64-linux-static/tools/gdal/gdalinfo --version
GDAL 3.8.3, released 2024/01/04
$ ldd vcpkg/installed/x64-linux-static/tools/gdal/gdalinfo
not a dynamic executable
Victory! Now you can hopefully add any other GDAL features/dependencies via
the gdal[core,tools,...] vcpkg feature options. *Fully* static binaries
aren't really a normal tested output, so you might run into further issues.
Hope that helps,
Rob :)
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
> > The directory './vcpkg/packages/_x64-linux' now contains the > > compiled static versions of the libraries. But where can I find the GDAL > > apps? These are not contained in this directory. Ignore packages. It is a staging area. `installed` is the real thing. Kai ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
Hi Michael, On Tue, 20 Feb 2024 at 11:22, Michael Otto wrote: > The directory './vcpkg/packages/_x64-linux' now contains the > compiled static versions of the libraries. But where can I find the GDAL > apps? These are not contained in this directory. Great! So if you change your vcpkg installation to use `gdal[tools]` instead of `gdal`, it will build the applications too. If you need other dependencies, see https://vcpkg.link/ports/gdal then you can install eg: `gdal[tools,curl]` to add curl too. The options shown under `default-features` are automatically installed. Rob :) > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
Hi Michael, On Tue, 13 Feb 2024 at 07:18, Michael Otto wrote: > at the moment it's only about the Linux platform. And yes, it should be a > static compilation with all the necessary dependencies (including libtiff > and all the others). > > I currently use an Ubuntu VM and use an Alpine-Linux Docker container for > compiling. First a static Geos library is created via cmake, then a static > Proj library (without Curl, because this currently leads to errors) and > then Gdal as a static library. > The basis of the script comes from this source: > https://github.com/OSGeo/gdal/issues/4815. > I have updated the version used to the current version and replaced the > existing './configure' and 'make' commands with cmake. > > This is what it looks like: Thanks, that's helpful :-) Following your script, there's no dynamic libgdal linked into the app binaries... # ldd /tmp/mo_gdal_install/bin/gdal_translate /lib/ld-musl-aarch64.so.1 (0xb968b000) libz.so.1 => /lib/libz.so.1 (0xb7e06000) libtiff.so.5 => /usr/lib/libtiff.so.5 (0xb7d87000) libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0xb7c8) libjpeg.so.8 => /usr/lib/libjpeg.so.8 (0xb7c2a000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7a6a000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xb7a45000) libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0xb968b000) If you want it to error instead of creating anything dynamic, add -DCMAKE_ EXE_LINKER_FLAGS=-static to your GDAL configuration. Then you get errors! [ 95%] Linking CXX executable test_ogrsf /usr/lib/gcc/aarch64-alpine-linux-musl/10.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: attempted static link of dynamic object `/lib/libz.so' collect2: error: ld returned 1 exit status make[2]: *** [apps/CMakeFiles/test_ogrsf.dir/build.make:112: apps/test_ogrsf] Error 1 make[1]: *** [CMakeFiles/Makefile2:8116: apps/CMakeFiles/test_ogrsf.dir/all] Error 2 make: *** [Makefile:149: all] Error 2 So you need to add CMake options to specifically point to the static libraries via -DJPEG_LIBRARY=/usr/lib/libjpegturbo.a and so on. Using the GDAL internal zlib/libtiff/etc build options might make your life simpler. Note that GDAL is pretty big! *Each* of gdaltranslate, gdalwarp, etc will end up at >30MB ... Rob :) ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
You may find the listings used to build static libraries for R Windows packages using MXE as the basis to be helpful: https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/toolchain_libs/mxe/src/ These are targeted at Windows, but the dependency tree logic should be similar. In building the successive static libraries, very likely the link order will matter too, so some patience is needed. Hope this helps, Roger -- Roger Bivand Emeritus Professor Norwegian School of Economics Postboks 3490 Ytre Sandviken, 5045 Bergen, Norway roger.biv...@nhh.no ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps
Hi Michael, On Mon, 12 Feb 2024 at 12:02, Michael Otto via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > > The goal is to cast GDAL and all its dependencies (PROJ / GEOS / all > dependencies to system libraries / ...) into a static library and to create > the GDAL apps as static executable programs. Which platform are you working on? If you need a fully static compilation of everything *all* the way down including openssl + zlib + zstd + libtiff + all the other libraries, using vcpkg[1] might be a reasonable approach. By default it does static compilation on macOS & Linux, and it has a static compilation option on Windows (the default on Windows is dynamic). It doesn't support every compile option though, but it's customisable and if you're after a reasonably vanilla GDAL it should work ok. > > There should be no dynamic dependencies. Note that GDAL + Proj (maybe others), have data files they rely on, so while you may be able to get a single executable, you'll still need to distribute these files. I can imagine ways around that, but they're getting fairly complex. > Unfortunately, I have not had any success so far. The library is created > statically but the apps are not yet. > Does anyone have experience with this topic or possibly a procedure that > leads to success? Can you share the process you've currently got to? Rob :) [1] https://github.com/microsoft/vcpkg ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Build static GDAL-Lib and static GDAL-Apps
Hello, I have been trying for some time to compile GDAL and the GDAL apps as a static library / executable programs. The goal is to cast GDAL and all its dependencies (PROJ / GEOS / all dependencies to system libraries / ...) into a static library and to create the GDAL apps as static executable programs. There should be no dynamic dependencies. Unfortunately, I have not had any success so far. The library is created statically but the apps are not yet. Does anyone have experience with this topic or possibly a procedure that leads to success? Best regards. Michael Otto ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
