Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- Guy Harris via tcpdump-workers wrote: > (The existence of libtool is an indication that shared libraries have > gotten messy on UN*X.) > Perhaps for this particular case the right thing to do is to set > LD_LIBRARY_PATH when running the temporarily-installed tcpdump. Effectively, this is what libtool tries to do. I would rather just be explicit about it somehow. Maybe that goes into how we use "make check", but I'm not sure where else it matters. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ --- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- On Jan 22, 2021, at 7:11 PM, Guy Harris via tcpdump-workers wrote: > I'll try experimenting with one of my Ubuntu VMs. Welcome to Shared Library Search Hell. Most UN*Xes have a notion of RPATH (with, of course, different compiler command-line flags to set it). pcap-config provides one if the shared library isn't going to be installed in /usr/lib. The pkg-config file doesn't provide one, however, and some searching indicates that the pkg-config maintainers recommend *against* doing so. They recommend using libtool when linking, instead. Part of the problem here may be that setting the RPATH in an executable affects how it searches for *all* libraries, so it could affect which version of an unrelated library is found. (The existence of libtool is an indication that shared libraries have gotten messy on UN*X.) Perhaps for this particular case the right thing to do is to set LD_LIBRARY_PATH when running the temporarily-installed tcpdump. The macOS linker appears to put absolute paths for shared libraries into the executable by default: $ otool -L /bin/cat /bin/cat: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1) so this may not be an issue there. (Also, the existence of the term "DLL hell" is an indication that shared libraries have gotten messy on Windows, but I digress :-))--- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- Guy Harris via tcpdump-workers wrote: >> $ /tmp/libpcap/bin/pcap-config --libs -L/tmp/libpcap/lib >> -Wl,-rpath,/tmp/libpcap/lib -lpcap > So that *should* cause /tmp/libpcap/lib to be added to the executable's > path, which *should* cause it to look in /tmp/libpcap/lib for shared > libraries. I have never seen this work since the move from SunOS-style shared-library/AOUT to ELF twenty years ago. -R/tmp/libpcap/lib is what I used to do, but AFAIK, this is equally a no-op. > I'll try experimenting with one of my Ubuntu VMs. > In the meantime, for some fun head-exploding reading, take a look at > https://en.wikipedia.org/wiki/Rpath > and perhaps some other documents found by a search for Yeah... I don't even know what to say. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ --- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- On Jan 22, 2021, at 2:54 PM, Denis Ovsienko via tcpdump-workers wrote: > I have tested it again with the current master branches of libpcap and > tcpdump. Both builds (with and without libpcap0.8-dev) now complete > without errors. > > However, in both cases the installed tcpdump fails to run because it > is linked with libpcap.so.1. Which, as far as I can remember, > previously somehow managed to resolve to the > existing /tmp/libpcap/lib/libpcap.so.1, but not amymore: > > $ /tmp/libpcap/bin/tcpdump --version > /tmp/libpcap/bin/tcpdump: error while loading shared libraries: > libpcap.so.1: cannot open shared object file: No such file or directory > > $ ldd /tmp/libpcap/bin/tcpdump > linux-vdso.so.1 (0x7ffdc7ffe000) > libpcap.so.1 => not found > libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 > (0x7f34522ac000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 > (0x7f3451ebb000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 > (0x7f3451cb7000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x7f3451a98000) > /lib64/ld-linux-x86-64.so.2 (0x7f3452c6f000) > > $ /tmp/libpcap/bin/pcap-config --libs > -L/tmp/libpcap/lib -Wl,-rpath,/tmp/libpcap/lib -lpcap So that *should* cause /tmp/libpcap/lib to be added to the executable's path, which *should* cause it to look in /tmp/libpcap/lib for shared libraries. So, if there's a /tmp/libpcap/lib/libpcap.so.1 file, that's not happening, somehow. I'll try experimenting with one of my Ubuntu VMs. In the meantime, for some fun head-exploding reading, take a look at https://en.wikipedia.org/wiki/Rpath and perhaps some other documents found by a search for lpath rpath linux--- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- On Wed, 9 Sep 2020 17:07:25 +0100 Denis Ovsienko via tcpdump-workers wrote: > Here are my steps to reproduce: > > libpcap$ ./configure --enable-remote --prefix=/tmp/libpcap > libpcap$ make > libpcap$ make install > tcpdumpbuild$ cmake -DWITH_CRYPTO="yes" > -DCMAKE_PREFIX_PATH=/tmp/libpcap -DCMAKE_INSTALL_PREFIX=/tmp/libpcap > /path/to/tcpdump_git_clone > > The above fails if the package libpcap0.8-dev is installed. After > removing the package the above works and tcpdump links > with /tmp/libpcap/lib/libpcap.so.1 as expected. I have tested it again with the current master branches of libpcap and tcpdump. Both builds (with and without libpcap0.8-dev) now complete without errors. However, in both cases the installed tcpdump fails to run because it is linked with libpcap.so.1. Which, as far as I can remember, previously somehow managed to resolve to the existing /tmp/libpcap/lib/libpcap.so.1, but not amymore: $ /tmp/libpcap/bin/tcpdump --version /tmp/libpcap/bin/tcpdump: error while loading shared libraries: libpcap.so.1: cannot open shared object file: No such file or directory $ ldd /tmp/libpcap/bin/tcpdump linux-vdso.so.1 (0x7ffdc7ffe000) libpcap.so.1 => not found libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x7f34522ac000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f3451ebb000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f3451cb7000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f3451a98000) /lib64/ld-linux-x86-64.so.2 (0x7f3452c6f000) $ /tmp/libpcap/bin/pcap-config --libs -L/tmp/libpcap/lib -Wl,-rpath,/tmp/libpcap/lib -lpcap It may be a user error, or there may be some other edge case hidden in the combination of the two build systems. So long as it does not cause obvious failures to build, I would be fine to leave it as it is now. -- Denis Ovsienko --- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- On Jan 7, 2021, at 5:41 PM, Denis Ovsienko via tcpdump-workers wrote: > 5 years of backward compatibility might be OK'ish, although from time > to time I run into such "long-term support" systems that in practice > mean someone keeps paying good money for "support" for 5-10 years, but > they don't get bugs fixed or new software versions backported. In my > own experience this tends to have something to do with RedHat Linux > distributions. Yeah, a lot of people are running old RHEL or CentOS - a lot of them keep asking about newer versions of Wireshark, because the older RHEL/CentOS versions provide Wireshark 2.x packages when Wireshark's already up to 3.4.2. They probably provide old tcpdump, too, so people might want to build newer versions. I've checked in a change that picks up some code from a newer version of the pkg-config CMake module to make it work with older versions.--- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- On Thu, 7 Jan 2021 16:46:41 -0800 Guy Harris wrote: > 3.1 dates back to 2015. That might be sufficient to treat as a > minimum. 5 years of backward compatibility might be OK'ish, although from time to time I run into such "long-term support" systems that in practice mean someone keeps paying good money for "support" for 5-10 years, but they don't get bugs fixed or new software versions backported. In my own experience this tends to have something to do with RedHat Linux distributions. Regarding the solution you consider, I have not tried the fixes from your previous message yet and I do not understand every facet of the problem. Have you considered to declare somewhere that certain ways of building work only in this and this particular basic manner? And/or to disable some smart ways of libpcap detection automatically if cmake is too old, instead of failing the build outright? -- Denis Ovsienko --- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- On Jan 7, 2021, at 3:21 PM, Guy Harris via tcpdump-workers wrote: > So we should either 1) require CMake 3.1 or later or 2) forcibly set > PKG_CONFIG_USE_CMAKE_PREFIX_PATH to YES. For now, my inclination is to do > the latter. That won't work - PKG_CONFIG_USE_CMAKE_PREFIX_PATH *isn't supported* prior to 3.1. 3.1 dates back to 2015. That might be sufficient to treat as a minimum.--- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message ---
On Sep 9, 2020, at 9:07 AM, Denis Ovsienko via tcpdump-workers
wrote:
> The "Found pcap-config" message means that FindPCAP.cmake has not found
> libpcap by means of pkg-config, and the lack of the message means the
> pkg-config method worked. A few weeks ago Ubuntu 18.04 systems started
> to have the libpcap.pc file in the libpcap0.8-dev package, so on such a
> system "pkg-config --libs libpcap" now prints "-lpcap" and "pkg-config
> --cflags libpcap" prints an empty string, which makes sense.
It makes sense if you want to build with the *system* libpcap.
It does *not* make sense if you want to build with the libpcap that was built,
and installed under /tmp, in the Travis build. For *that*, you'd want
"pkg-config --libs libpcap" to print "-L/tmp/lib -lpcap" and you'd want
pkg-config --cflags libpcap" to print "-I /tmp/include".
That's because it's finding the libpcap.pc file in the libpcap0.8-dev package,
not the one in /tmp/lib/pkgconfig; the latter one should do what we want.
I changed .travis.yml to run CMake with PKG_CONFIG_PATH=/tmp/lib/pkgconfig,
which appears to make it find the right .pc file and thus to find the right
libpcap.
> What does not make sense is that CMake seems to use non-pkg-config
> flags to tell if a specific feature is available,
It *should* be using the pkg-config flags - the code that tests for features
just does
#
# libpcap/WinPcap/Npcap.
# First, find it.
#
find_package(PCAP REQUIRED)
include_directories(${PCAP_INCLUDE_DIRS})
cmake_push_check_state()
#
# Now check headers.
#
set(CMAKE_REQUIRED_INCLUDES ${PCAP_INCLUDE_DIRS})
#
# Check whether we have pcap/pcap-inttypes.h.
# If we do, we use that to get the C99 types defined.
#
check_include_file(pcap/pcap-inttypes.h HAVE_PCAP_PCAP_INTTYPES_H)
#
# Check for various functions in libpcap/WinPcap/Npcap.
#
cmake_push_check_state()
set(CMAKE_REQUIRED_LIBRARIES ${PCAP_LIBRARIES})
#
# Check for "pcap_list_datalinks()" and use a substitute version if
# it's not present. If it is present, check for
"pcap_free_datalinks()";
# if it's not present, we don't replace it for now. (We could do so
# on UN*X, but not on Windows, where hilarity ensues if a program
# built with one version of the MSVC support library tries to free
# something allocated by a library built with another version of
# the MSVC support library.)
#
check_function_exists(pcap_list_datalinks HAVE_PCAP_LIST_DATALINKS)
...
cmake_pop_check_state()
which doesn't care whether PCAP_INCLUDE_DIRS and PCAP_LIBRARIES were set from
pkg-config or pcap-config or manually poking the system.
> but uses pkg-config
> flags to compile and link the source,
*However*, the CMake documentation says about CMAKE_PREFIX_PATH:
Semicolon-separated list of directories specifying installation
prefixes to be searched by the find_package(), find_program(), find_library(),
find_file(), and find_path() commands.
and we're setting CMAKE_PREFIX_PATH, so if any of the include or library checks
use CMAKE_PREFIX_PATH, then they might find headers for the libpcap installed
in /tmp/libpcap, even though the build itself will use flags from the system
libpcap.pc.
The CMake 3.19 documentation for FindPkgConfig, which is the module for using
pkg-config:
https://cmake.org/cmake/help/v3.19/module/FindPkgConfig.html
says:
PKG_CONFIG_USE_CMAKE_PREFIX_PATH
Specifies whether pkg_check_modules() and pkg_search_module()
should add the paths in the CMAKE_PREFIX_PATH, CMAKE_FRAMEWORK_PATH and
CMAKE_APPBUNDLE_PATH cache and environment variables to thepkg-config search
path.
If this variable is not set, this behavior is enabled by
default if CMAKE_MINIMUM_REQUIRED_VERSION is 3.1 or later, disabled otherwise.
So we should either 1) require CMake 3.1 or later or 2) forcibly set
PKG_CONFIG_USE_CMAKE_PREFIX_PATH to YES. For now, my inclination is to do the
latter.
Once all the other stuff I've checked in passes Travis, I'll try that instead
of explicitly setting PKG_CONFIG_PATH. and see if that works.
> and when the system has one
> libpcap version installed as a package and another version that the
> user wants to build with, that very easily breaks (and even if it does
> not, the end result is not what the user was expecting).
>
> Here are my steps to reproduce:
>
> libpcap$ ./configure --enable-remote --prefix=/tmp/libpcap
> libpcap$ make
> libpcap$ make install
> tcpdumpbuild$ cmake -DWITH_CRYPTO="yes"
> -DCMAKE_PREFIX_PATH=/tmp/libpcap -DCMAKE_INSTALL_PREFIX=/tmp/libpcap
> /path/to/tcpdump_git_clone
Try that with
PKG_CONFIG_PATH=/tmp/libpcap/lib/pkgconfig cmake -DWITH_CRYPTO="yes"
-DCMAKE_PREFIX_PATH=/tmp/libpcap -DC
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- On Jan 7, 2021, at 9:35 AM, Bill Fenner via tcpdump-workers wrote: > These jobs are still failing, but now for a different reason. Part of the problem is that pkg-config wasn't finding the locally-installed libpcap under /tmp, because PKG_CONFIG_PATH wasn't set to point to /tmp/lib/pkgconfig. We're now doing that, and I re-enabled those jobs; so far, the GCC builds on Linux seem to be working for BUILD_LIBPCAP=yes CMAKE=yes.--- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- On 07/01/2021 18:35, Bill Fenner via tcpdump-workers wrote: > These jobs are still failing, but now for a different reason. The build > succeeds, but the tests fail - the tests that require the new libpcap. > However, if you augment TESTrun to print the version, it says > 1.11.0-PRE-GIT but the tests that need the new libpcap still fail. My hypothesis is that, with the cmake build, for some reason, tcpdump linked to the 1.11.0-PRE-GIT local library was built with include files for the system libpcap not the local libpcap ones, thus DLT_DSA_TAG_BRCM and some other macros (which exist only in the new libpcap) are undefined (hence the UNSUPPORTED outputs). '-I' configuration problem? (The autoconf build add in the Makefile: INCLS = -I. -I../libpcap) -- Francois-Xavier --- End Message --- ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message ---
On Wed, Sep 9, 2020 at 12:08 PM Denis Ovsienko via tcpdump-workers <
tcpdump-workers@lists.tcpdump.org> wrote:
> Travis CI tcpdump builds have been failing for a while and I went to
> see why. It is easy to see that only the jobs that have
> "BUILD_LIBPCAP=yes CMAKE=yes" fail
These jobs are still failing, but now for a different reason. The build
succeeds, but the tests fail - the tests that require the new libpcap.
However, if you augment TESTrun to print the version, it says
1.11.0-PRE-GIT but the tests that need the new libpcap still fail. I can
not find anything that changes LD_LIBRARY_PATH, or anything, but shortly
after running
print "Running tests from ${testsdir}\n";
print "using ${TCPDUMP}\n";
system "${TCPDUMP} --version";
we run
$r = system "$TCPDUMP -# -n -r $input $options >tests/NEW/${outputbase}
2>${rawstderrlog}";
and that uses the wrong libpcap.
I am at loose ends as to how to debug further - a sample log from my
attempt to see what's going on is
https://api.travis-ci.com/v3/job/468210839/log.txt - the diffs in that run
are very simple (and the env. variable one is probably
incorrect/unnecessary) -
https://github.com/the-tcpdump-group/tcpdump/compare/master...fenner:cmake-pcap
Bill
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
[tcpdump-workers] libpcap detection and linking in tcpdump
--- Begin Message --- Hello list. Travis CI tcpdump builds have been failing for a while and I went to see why. It is easy to see that only the jobs that have "BUILD_LIBPCAP=yes CMAKE=yes" fail, for example [1], and the same jobs built well before, for example [2]. The tcpdump build process fails in these cases because CMake detects the presence and usability of particular libpcap elements before the compilation, but during the compilation the compiler is not finding these elements. For example: tcpdump.c:1251:19: error: ‘PCAP_OPENFLAG_PROMISCUOUS’ undeclared (first use in this function); did you mean ‘PCAP_WARNING_PROMISC_NOTSUP’? (The above compiles only when HAVE_PCAP_OPEN is defined.) I have spent some time trying to reproduce the amd64 Linux GCC case on my desktop. Initially I though it was a regression in libpcap because tcpdump had no meaningful changes between the successful and the failed jobs, and the job always clones the current master branch of libpcap. As it turned out, the reason for failure was not because of a change in tcpdump or libpcap, but because of a change in the Travis CI build environment, which is Ubuntu 18.04 and which recently started to include a pkg-config file in the libpcap package: libpcap (1.8.1-6ubuntu1.18.04.2) bionic; urgency=medium * Install pkg-config file for libpcap (LP: #1865501). - d/p/set-package-name-with-ac-init.patch: Set the PACKAGE_NAME variable using autoconf's AC_INIT macro, so that we can use it in the libpcap.pc.in file. - d/p/use-m4-macro-to-get-version.patch: Use an M4 macro to execute a "cat" command and obtain the version from the VERSION file. - d/p/install-pkg-config-file.patch: New patch from upstream, which creates a libpcap.pc.in file and adjusts the Makefile to install it. - d/libpcap0.8-dev.install: Install libpcap.pc pkg-config file. Thanks to Luca Boccassi for the Debian patch (Closes #922219). -- Sergio Durigan Junior Mon, 29 Jun 2020 17:49:04 -0400 If you examine the provided example jobs closely (the "Raw log" button returns the full output in plain text), here is the difference in CMake output when the job changes from good to bad: @@ -71,9 +71,8 @@ -- Check size of struct in6_addr - done -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Checking for one of the modules 'libpcap' --- Found pcap-config -- Looking for pcap/pcap-inttypes.h --- Looking for pcap/pcap-inttypes.h - found +-- Looking for pcap/pcap-inttypes.h - not found -- Looking for pcap_list_datalinks -- Looking for pcap_list_datalinks - found -- Looking for pcap_free_datalinks The "Found pcap-config" message means that FindPCAP.cmake has not found libpcap by means of pkg-config, and the lack of the message means the pkg-config method worked. A few weeks ago Ubuntu 18.04 systems started to have the libpcap.pc file in the libpcap0.8-dev package, so on such a system "pkg-config --libs libpcap" now prints "-lpcap" and "pkg-config --cflags libpcap" prints an empty string, which makes sense. What does not make sense is that CMake seems to use non-pkg-config flags to tell if a specific feature is available, but uses pkg-config flags to compile and link the source, and when the system has one libpcap version installed as a package and another version that the user wants to build with, that very easily breaks (and even if it does not, the end result is not what the user was expecting). Here are my steps to reproduce: libpcap$ ./configure --enable-remote --prefix=/tmp/libpcap libpcap$ make libpcap$ make install tcpdumpbuild$ cmake -DWITH_CRYPTO="yes" -DCMAKE_PREFIX_PATH=/tmp/libpcap -DCMAKE_INSTALL_PREFIX=/tmp/libpcap /path/to/tcpdump_git_clone The above fails if the package libpcap0.8-dev is installed. After removing the package the above works and tcpdump links with /tmp/libpcap/lib/libpcap.so.1 as expected. Clearly, there are cases where using pkg-config would be the right thing to do. But is seems wrong to try it always as the first choice. Perhaps it would be better to try pkg-config after the user-specified prefix did not result in a usable libpcap. Autoconf is not entirely consistent with CMake, in that it tries pkg-config before libpcap in the specified prefix directory too, but there are differences: * Before these two methods autoconf checks for libpcap in ../libpcap and links it statically if it is available. CMake does not do that. * If the above was not the case, the pkg-config method works in autoconf, so tcpdump is linked with the system libpcap. CMake build breaks in this case. * If the above was not the case and ./configure is given a prefix, it fails to find libpcap under that prefix. CMake gets that right. To me right now it looks as if the best first thing to do would be fixing pkg-config in CMake, that should recover Travis CI. Then the next best thing to do would be reordering the steps in both autoconf and CMake to try pkg-config after pcap-confi
