Re: How to build compatible packages that use Eigen?
Ryo IGARASHI writes: > Looking at your patch, why do you delete all the #ifndef block in the > first place? Adding just "#define EIGEN_MALLOC_ALREADY_ALIGNED 0" line > on top of the #ifndef block should do the job. That would work, but I don't think it makes a huge difference either way. The bigger patch is maybe a bit safer against future changes: I can imagine upstream changing/adding some logic to that big block that would re-define EIGEN_MALLOC_ALREADY_ALIGNED further down, making the one-line patch ineffective. In that situation the bigger patch would create a merge conflict, requiring human review, and the human would hopefully catch that. But again, I don't care a whole lot one way or the other
Re: How to build compatible packages that use Eigen?
Hi, Dima, Looking at your patch, why do you delete all the #ifndef block in the first place? Adding just "#define EIGEN_MALLOC_ALREADY_ALIGNED 0" line on top of the #ifndef block should do the job. 2024年2月7日(水) 16:56 Dima Kogan : > > I hit this problem again at work yesterday, so let's fix it. > > I'm attaching a patch that can be added to the quilt patch stack in our > libeigen3-dev package. This disables the conditional preprocessor logic > that selects the allocator/deallocator behavior. With this patch we > always make no assumptions about the alignment of malloc(), and we > ALWAYS use eigen's manual allocator to force alignment. This is maybe a > bit less efficient than the other path, but dynamic allocation shouldn't > be in any hot code path anyway. > > We could also use aligned_alloc() or memalign() instead here, also > unconditionally. The "unconditional" is the important part: it must be > consistent with any build flags, otherwise stuff can crash. > > Is this reasonable? > > Thanks. > Best regards, -- Ryo IGARASHI, Ph.D. rigar...@gmail.com
Re: How to build compatible packages that use Eigen?
I hit this problem again at work yesterday, so let's fix it.
I'm attaching a patch that can be added to the quilt patch stack in our
libeigen3-dev package. This disables the conditional preprocessor logic
that selects the allocator/deallocator behavior. With this patch we
always make no assumptions about the alignment of malloc(), and we
ALWAYS use eigen's manual allocator to force alignment. This is maybe a
bit less efficient than the other path, but dynamic allocation shouldn't
be in any hot code path anyway.
We could also use aligned_alloc() or memalign() instead here, also
unconditionally. The "unconditional" is the important part: it must be
consistent with any build flags, otherwise stuff can crash.
Is this reasonable?
Thanks.
Description: https://www.mail-archive.com/debian-science@lists.debian.org/msg13666.html
This patch ensures that Eigen memory allocation/free are implemented the same
way REGARDLESS of build flags. This ensures that an application using Eigen
(possibly indirectly, through libraries) will not crash due to non-identical
build flags
Author: Dima Kogan
diff --git a/Eigen/src/Core/util/Memory.h b/Eigen/src/Core/util/Memory.h
index 875318c..38d55d1 100644
--- a/Eigen/src/Core/util/Memory.h
+++ b/Eigen/src/Core/util/Memory.h
@@ -17,50 +17,13 @@
*** Platform checks for aligned malloc functions ***
*/
#ifndef EIGEN_MEMORY_H
#define EIGEN_MEMORY_H
-#ifndef EIGEN_MALLOC_ALREADY_ALIGNED
-
-// Try to determine automatically if malloc is already aligned.
-
-// On 64-bit systems, glibc's malloc returns 16-byte-aligned pointers, see:
-// http://www.gnu.org/s/libc/manual/html_node/Aligned-Memory-Blocks.html
-// This is true at least since glibc 2.8.
-// This leaves the question how to detect 64-bit. According to this document,
-// http://gcc.fyxm.net/summit/2003/Porting%20to%2064%20bit.pdf
-// page 114, "[The] LP64 model [...] is used by all 64-bit UNIX ports" so it's indeed
-// quite safe, at least within the context of glibc, to equate 64-bit with LP64.
-#if defined(__GLIBC__) && ((__GLIBC__>=2 && __GLIBC_MINOR__ >= 8) || __GLIBC__>2) \
- && defined(__LP64__) && ! defined( __SANITIZE_ADDRESS__ ) && (EIGEN_DEFAULT_ALIGN_BYTES == 16)
- #define EIGEN_GLIBC_MALLOC_ALREADY_ALIGNED 1
-#else
- #define EIGEN_GLIBC_MALLOC_ALREADY_ALIGNED 0
-#endif
-
-// FreeBSD 6 seems to have 16-byte aligned malloc
-// See http://svn.freebsd.org/viewvc/base/stable/6/lib/libc/stdlib/malloc.c?view=markup
-// FreeBSD 7 seems to have 16-byte aligned malloc except on ARM and MIPS architectures
-// See http://svn.freebsd.org/viewvc/base/stable/7/lib/libc/stdlib/malloc.c?view=markup
-#if defined(__FreeBSD__) && !(EIGEN_ARCH_ARM || EIGEN_ARCH_MIPS) && (EIGEN_DEFAULT_ALIGN_BYTES == 16)
- #define EIGEN_FREEBSD_MALLOC_ALREADY_ALIGNED 1
-#else
- #define EIGEN_FREEBSD_MALLOC_ALREADY_ALIGNED 0
-#endif
-
-#if (EIGEN_OS_MAC && (EIGEN_DEFAULT_ALIGN_BYTES == 16)) \
- || (EIGEN_OS_WIN64 && (EIGEN_DEFAULT_ALIGN_BYTES == 16)) \
- || EIGEN_GLIBC_MALLOC_ALREADY_ALIGNED \
- || EIGEN_FREEBSD_MALLOC_ALREADY_ALIGNED
- #define EIGEN_MALLOC_ALREADY_ALIGNED 1
-#else
- #define EIGEN_MALLOC_ALREADY_ALIGNED 0
-#endif
-
-#endif
+#define EIGEN_MALLOC_ALREADY_ALIGNED 0
namespace Eigen {
namespace internal {
EIGEN_DEVICE_FUNC
Re: How to build compatible packages that use Eigen?
Hi Anton. Thanks for replying. Anton Gladky writes: > I have been maintaining the Eigen library in Debian for over 12 years, > and I cannot recall the specific bug ticket related to this topic. I believe it. I suspect the reason is that you get mystery crashes that require more-than-the-usual-amount-of knowledge to conclusively debug and then report the bug to the right place. In any case, once I figured out what was causing my problem, I see references to it all over the place. For instance: - https://eigen.tuxfamily.org/dox/TopicPreprocessorDirectives.html (search for "ABI") - https://stackoverflow.com/questions/50723121/eigen-crash-due-to-unaligned-access-with-msvc14-and-avx - https://eigen.tuxfamily.org/bz/show_bug.cgi?id=554 - https://stackoverflow.com/questions/71647793/eigen-static-lib-aligned-free-double-free-or-corruption Upstream should do better here, but I guess they decided they don't care. > However, it seems that the question you have raised is indeed valid. > If patching Eigen in those two places would help resolve the issue, > please prepare a patch, and I believe we can proceed with pushing it. OK. Let me construct the minimal program for the other crash of this kind I've seen (something about alignment), and then I'll propose some patches. > Does it make sense to discuss this with the Eigen developers? Or is > the question very specific to Debian (or packaging) and might not be > of interest to them? If I was them I wouldn't want my thing to crash. But there're references in their own docs and bug tracker that suggest that it's a known and accepted issue that mixed build flags make it die. Probably we can suggest the safe behavior behind and #ifdef at least. Let me make the patches so that we can have something to talk about. Thanks much.
Re: How to build compatible packages that use Eigen?
Hi Dima,
I have been maintaining the Eigen library in Debian for over 12 years,
and I cannot recall the specific bug ticket related to this topic. However,
it seems that the question you have raised is indeed valid. If patching
Eigen in those two places would help resolve the issue, please prepare
a patch, and I believe we can proceed with pushing it.
Does it make sense to discuss this with the Eigen developers? Or is the
question very specific to Debian (or packaging) and might not be of interest
to them?
Best regards,
Anton
Am Mi., 12. Juli 2023 um 01:24 Uhr schrieb Dima Kogan :
> Hi.
>
> Apologies for taking so long to look at this again; I've been busy.
>
> I just looked into it, and my conclusion is that there's no way to
> ensure that Eigen won't crash without us patching our package. So we
> should patch our package.
>
> I'm attaching a tiny demo program. Extract it and
>
> make && ./main
>
> You'll see that it crashes. We have lib.cc:
>
> #include
>
> __attribute__((visibility("default")))
> Eigen::MatrixXd* make_array()
> {
> return new Eigen::MatrixXd(10,20);
> }
>
> This is an analogue of some library we would be packaging for Debian.
> This would be built with no cpu-specific options, which is what the
> Makefile in this demo does.
>
> We also have a main.cc:
>
> #include
>
> Eigen::MatrixXd* make_array();
> int main(void)
> {
> Eigen::MatrixXd* matrix = make_array();
> delete matrix;
> return 0;
> }
>
> This is an analogue of some user program that uses this library. This
> isn't going into Debian, and the user wants to use their CPU fully, so
> they build with -mavx. This causes Eigen to crash. Because the
> allocation and deallocation paths don't work the same.
>
> In this demo no Eigen symbols are exported from lib.so, so it's not a
> case of the linker picking the wrong weak symbol, and this cannot be
> fixed by symbol versioning or visibility settings or anything.
>
> This isn't a contrived example. I hit this in the real-world with a
> package I'm going to upload soon (g2o).
>
> Can anybody see ways to make Eigen work reliably here without patching
> away the different paths in aligned_malloc() and aligned_free() ?
>
>
> https://sources.debian.org/src/eigen3/3.4.0-4/Eigen/src/Core/util/Memory.h/#L179
>
> https://sources.debian.org/src/eigen3/3.4.0-4/Eigen/src/Core/util/Memory.h/#L200
>
> I don't see any way to do it currently, and probably we should patch
> these out.
>
> There was also a second similar problem I saw earlier, that resulted in
> crashes due to inconsistent alignment. I'm going to revisit that
> shortly.
>
> Thanks.
>
>
Re: How to build compatible packages that use Eigen?
Hi.
Apologies for taking so long to look at this again; I've been busy.
I just looked into it, and my conclusion is that there's no way to
ensure that Eigen won't crash without us patching our package. So we
should patch our package.
I'm attaching a tiny demo program. Extract it and
make && ./main
You'll see that it crashes. We have lib.cc:
#include
__attribute__((visibility("default")))
Eigen::MatrixXd* make_array()
{
return new Eigen::MatrixXd(10,20);
}
This is an analogue of some library we would be packaging for Debian.
This would be built with no cpu-specific options, which is what the
Makefile in this demo does.
We also have a main.cc:
#include
Eigen::MatrixXd* make_array();
int main(void)
{
Eigen::MatrixXd* matrix = make_array();
delete matrix;
return 0;
}
This is an analogue of some user program that uses this library. This
isn't going into Debian, and the user wants to use their CPU fully, so
they build with -mavx. This causes Eigen to crash. Because the
allocation and deallocation paths don't work the same.
In this demo no Eigen symbols are exported from lib.so, so it's not a
case of the linker picking the wrong weak symbol, and this cannot be
fixed by symbol versioning or visibility settings or anything.
This isn't a contrived example. I hit this in the real-world with a
package I'm going to upload soon (g2o).
Can anybody see ways to make Eigen work reliably here without patching
away the different paths in aligned_malloc() and aligned_free() ?
https://sources.debian.org/src/eigen3/3.4.0-4/Eigen/src/Core/util/Memory.h/#L179
https://sources.debian.org/src/eigen3/3.4.0-4/Eigen/src/Core/util/Memory.h/#L200
I don't see any way to do it currently, and probably we should patch
these out.
There was also a second similar problem I saw earlier, that resulted in
crashes due to inconsistent alignment. I'm going to revisit that
shortly.
Thanks.
eigen-weak-linking-bug.tar.gz
Description: application/gzip
Re: How to build compatible packages that use Eigen?
Thank you very much for the notes. The Eigen code I see does everything with the preprocessor. Which is fine, and doesn't mean that it must crash. For the specific package I'm building (libg2o) Eigen is the only option, but this sounds like a generic problem that affects all sorts of packages, not just libg2o. I'm a bit surprised that we don't have obvious bug reports related to this. There are lots of reports of crashes on various github pages, with the replies being "you must match build flags, especially -m". So this isn't a completely unknown problem. I've seen two kinds of crashes so far: - Mismatched alloc/free. This is described in the first email in this thread - Mismatched alignment. Something calls aligned_malloc() to get a pointer with some alignment. And some other Eigen function then ingests this pointer and asserts because it isn't aligned sufficiently (this second function was compiled differently). I've seen this happen here: https://sources.debian.org/src/eigen3/3.4.0-4/Eigen/src/Core/MapBase.h/#L198 A number of structures in Eigen are aligned to "AlignedMax", defined here: https://sources.debian.org/src/eigen3/3.4.0-4/Eigen/src/Core/util/Constants.h/#L241 Which eventually comes from EIGEN_IDEAL_MAX_ALIGN_BYTES, defined here: https://sources.debian.org/src/eigen3/3.4.0-4/Eigen/src/Core/util/ConfigureVectorization.h/#L57 I can imagine there are other crashes that we could get. And I'm also suspecting that simply limiting libg2o Eigen calls to implementations inside libg2o won't conclusively fix everything. Probably what we should do is: - patch the Eigen alloc/free functions to do a consistent thing - patch Eigen to set EIGEN_IDEAL_MAX_ALIGN_BYTES to a constant maximum I should run more experiments, though. Thanks
Re: How to build compatible packages that use Eigen?
I did something similar. Here is my example: The linker script is here: https://salsa.debian.org/science-team/blis/-/blob/master/debian/version_script.lds This script is used like this: https://salsa.debian.org/science-team/blis/-/blob/master/debian/patches/libblas-provider.patch This patch aims to hide some ABIs from the external calls. On Wed, 2023-05-03 at 22:07 -0400, Aaron M. Ucko wrote: > Dima Kogan writes: > > > Thanks for replying > > No problem. > > > Sorry, I'm not familiar-enough with linker scripts. I would pass this to > > the linker when building libg2o.so? Or the end-user would need to use > > this when build-time linking their application? The run-time dynamic > > linker doesn't need this, right? > > AIUI, you'd supply it when building libg2o.so itself, via the > --version-script linker option (or as an implict linker script, but I'd > favor being more explicit here). >
Re: How to build compatible packages that use Eigen?
Don't know how to address this issue but have some relevant comments.
Eigen library does not support run time dispatch for CPU ISAs.
When a package is built upon the amd64 baseline ISA but ran on a modern
CPU, the performance can be very poor.
This is why I build the tensorflow and pytorch packages against
libblas.so.3 (through update-alternatives).
A good BLAS implementation is usually faster than the Eigen
compiled in native ISA. For example, openblas.
Check if the library in question supports building against BLAS/LAPACK
instead of Eigen. Good luck if the upstream does not support that.
By the way, since there is no runtime ISA dispatch, the "-mavx" flag
is likely a baseline violation with RC severity.
I don't know whether Eigen implemented the dispatch in the latest
version. But the current state of this library still seems to dispatch
by preprocessor.
On Wed, 2023-05-03 at 14:18 -0700, Dima Kogan wrote:
> Hi. I'm packaging something that uses Eigen, and I'm running into a
> persistent compatibility issue I don't currently know how to solve. Help
> appreciated.
>
> Here's the problem. Eigen is a C++ header-only library that's heavy into
> templating. So all the functions inside Eigen produce weak symbols, and
> usually the linker will see many identical copies of the same weak
> symbol, from each compile unit and shared object being linked. The
> linker picks ONE of the weak definitions. This is the intended behavior
> in C++ because every copy is supposed to be identical. But in Eigen
> they're not identical: it does different things based on preprocessor
> defines, and you get crashes.
>
> Here's a simplified illustration of what happens.
>
>
> eigen3/Eigen/src/Core/util/Memory.h contains:
>
> EIGEN_DEVICE_FUNC inline void* aligned_malloc(std::size_t size)
> {
> check_that_malloc_is_allowed();
>
> void *result;
> #if (EIGEN_DEFAULT_ALIGN_BYTES==0) || EIGEN_MALLOC_ALREADY_ALIGNED
>
> EIGEN_USING_STD(malloc)
> result = malloc(size);
>
> #if EIGEN_DEFAULT_ALIGN_BYTES==16
> eigen_assert((size<16 || (std::size_t(result)%16)==0) && "System's
> malloc returned an unaligned pointer. Compile with
> EIGEN_MALLOC_ALREADY_ALIGNED=0 to fallback to handmade aligned memory
> allocator.");
> #endif
> #else
> result = handmade_aligned_malloc(size);
> #endif
>
> if(!result && size)
> throw_std_bad_alloc();
>
> return result;
> }
>
> EIGEN_DEVICE_FUNC inline void aligned_free(void *ptr)
> {
> #if (EIGEN_DEFAULT_ALIGN_BYTES==0) || EIGEN_MALLOC_ALREADY_ALIGNED
>
> EIGEN_USING_STD(free)
> free(ptr);
>
> #else
> handmade_aligned_free(ptr);
> #endif
> }
>
> The EIGEN_DEFAULT_ALIGN_BYTES and EIGEN_MALLOC_ALREADY_ALIGNED macros
> can vary based on things like __SSE__ and __AVX__ and such.
>
> Now let's say you're packaging a library. Let's call it libg2o. This is
> NOT header-only, and somewhere it does #include which
> eventually includes Memory.h. The libg2o.so that ends up in the
> "libg2o0" package then gets a weak symbol for "aligned_malloc" and
> "aligned_free" that encodes the compiler flags that were used when
> building libg2o.so.
>
> So far so good.
>
> Now let's say you have a user. They're writing a program that uses both
> libg2o and Eigen. They're writing their own application, not intended to
> go into Debian. So they build with -msse -mavx and all the other fun
> stuff. THEIR weak copies of "aligned_malloc" and "aligned_free" are
> different and incompatible with the copies in libg2o. And the
> application is then likely to crash because at least something somewhere
> will be allocated with one copy and deallocated with another.
>
> This is just terrible design from the eigen and c++ people, but that's
> what we have. Has anybody here run into this? How does one build the
> libg2o package so that users don't crash their application when using
> it? I tried to demand maximum alignment in libg2o, which fixes some
> things but not all. Currently debugging to find a better solution, but I
> suspect somebody has already fought this.
>
> Thanks
>
Re: How to build compatible packages that use Eigen?
Dima Kogan writes: > Thanks for replying No problem. > Sorry, I'm not familiar-enough with linker scripts. I would pass this to > the linker when building libg2o.so? Or the end-user would need to use > this when build-time linking their application? The run-time dynamic > linker doesn't need this, right? AIUI, you'd supply it when building libg2o.so itself, via the --version-script linker option (or as an implict linker script, but I'd favor being more explicit here). -- Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org) http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu
Re: How to build compatible packages that use Eigen?
Thanks for replying
u...@debian.org (Aaron M. Ucko) writes:
> It might help to make Eigen's symbols local to your shared library; to
> that end, I think it would work to feed the linker a version script
> reading something along the lines of
>
> {
> local:
> extern "C++" {
> Eigen::*;
> };
> global:
> *;
> };
Sorry, I'm not familiar-enough with linker scripts. I would pass this to
the linker when building libg2o.so? Or the end-user would need to use
this when build-time linking their application? The run-time dynamic
linker doesn't need this, right?
I'm pretty sure having libg2o only call its own copies of the Eigen
stuff would solve some of the problems, but I'm also pretty sure it will
not solve all of them in this case. But it's definitely worth trying.
Alternatively, we can patch Debian's copy of Eigen to remove all the
variability. That WOULD fix it, but I'd like to run more tests first.
Thanks
Re: How to build compatible packages that use Eigen?
Dima Kogan writes:
> Now let's say you have a user. They're writing a program that uses both
> libg2o and Eigen. They're writing their own application, not intended to
> go into Debian. So they build with -msse -mavx and all the other fun
> stuff. THEIR weak copies of "aligned_malloc" and "aligned_free" are
> different and incompatible with the copies in libg2o. And the
> application is then likely to crash because at least something somewhere
> will be allocated with one copy and deallocated with another.
Oy vey.
It might help to make Eigen's symbols local to your shared library; to
that end, I think it would work to feed the linker a version script
reading something along the lines of
{
local:
extern "C++" {
Eigen::*;
};
global:
*;
};
(or add such a local block to an existing version script).
--
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu
How to build compatible packages that use Eigen?
Hi. I'm packaging something that uses Eigen, and I'm running into a
persistent compatibility issue I don't currently know how to solve. Help
appreciated.
Here's the problem. Eigen is a C++ header-only library that's heavy into
templating. So all the functions inside Eigen produce weak symbols, and
usually the linker will see many identical copies of the same weak
symbol, from each compile unit and shared object being linked. The
linker picks ONE of the weak definitions. This is the intended behavior
in C++ because every copy is supposed to be identical. But in Eigen
they're not identical: it does different things based on preprocessor
defines, and you get crashes.
Here's a simplified illustration of what happens.
eigen3/Eigen/src/Core/util/Memory.h contains:
EIGEN_DEVICE_FUNC inline void* aligned_malloc(std::size_t size)
{
check_that_malloc_is_allowed();
void *result;
#if (EIGEN_DEFAULT_ALIGN_BYTES==0) || EIGEN_MALLOC_ALREADY_ALIGNED
EIGEN_USING_STD(malloc)
result = malloc(size);
#if EIGEN_DEFAULT_ALIGN_BYTES==16
eigen_assert((size<16 || (std::size_t(result)%16)==0) && "System's malloc
returned an unaligned pointer. Compile with EIGEN_MALLOC_ALREADY_ALIGNED=0 to
fallback to handmade aligned memory allocator.");
#endif
#else
result = handmade_aligned_malloc(size);
#endif
if(!result && size)
throw_std_bad_alloc();
return result;
}
EIGEN_DEVICE_FUNC inline void aligned_free(void *ptr)
{
#if (EIGEN_DEFAULT_ALIGN_BYTES==0) || EIGEN_MALLOC_ALREADY_ALIGNED
EIGEN_USING_STD(free)
free(ptr);
#else
handmade_aligned_free(ptr);
#endif
}
The EIGEN_DEFAULT_ALIGN_BYTES and EIGEN_MALLOC_ALREADY_ALIGNED macros
can vary based on things like __SSE__ and __AVX__ and such.
Now let's say you're packaging a library. Let's call it libg2o. This is
NOT header-only, and somewhere it does #include which
eventually includes Memory.h. The libg2o.so that ends up in the
"libg2o0" package then gets a weak symbol for "aligned_malloc" and
"aligned_free" that encodes the compiler flags that were used when
building libg2o.so.
So far so good.
Now let's say you have a user. They're writing a program that uses both
libg2o and Eigen. They're writing their own application, not intended to
go into Debian. So they build with -msse -mavx and all the other fun
stuff. THEIR weak copies of "aligned_malloc" and "aligned_free" are
different and incompatible with the copies in libg2o. And the
application is then likely to crash because at least something somewhere
will be allocated with one copy and deallocated with another.
This is just terrible design from the eigen and c++ people, but that's
what we have. Has anybody here run into this? How does one build the
libg2o package so that users don't crash their application when using
it? I tried to demand maximum alignment in libg2o, which fixes some
things but not all. Currently debugging to find a better solution, but I
suspect somebody has already fought this.
Thanks
