Re: Changes in HAProxy 3.0's Makefile and build options

[Илья Шипицин]

I'll postpone for a while.
I thought that value of "2" is the same as "1", I'll try to find better doc.

seems that I didn''t specify "march" and that might be the cause.

сб, 20 апр. 2024 г. в 15:21, Willy Tarreau :

> On Sat, Apr 20, 2024 at 03:11:19PM +0200,  ??? wrote:
> > ??, 20 ???. 2024 ?. ? 15:07, Willy Tarreau :
> >
> > > On Sat, Apr 20, 2024 at 02:49:38PM +0200,  ??? wrote:
> > > > ??, 11 ???. 2024 ?. ? 21:05, Willy Tarreau :
> > > >
> > > > > Hi Ilya,
> > > > >
> > > > > On Thu, Apr 11, 2024 at 08:27:39PM +0200,  ??? wrote:
> > > > > > do you know maybe how this was supposed to work ?
> > > > > > haproxy/Makefile at master · haproxy/haproxy (github.com)
> > > > > > 
> > > > >
> > > > > That's this:
> > > > >
> > > > >   ifneq ($(shell $(CC) $(CFLAGS) -dM -E -xc -  2>/dev/null |
> > > > > grep -c 'LOCK_FREE.*1'),0)
> > > > > USE_LIBATOMIC   = implicit
> > > > >   endif
> > > > >
> > > > > It calls the compiler with the known flags and checks if for this
> arch,
> > > > > it's configured to require libatomic.
> > > > >
> > > >
> > > > macros has changed from 1 to 2:
> > > >
> > > > ilia@fedora:~/Downloads$ cc -dM -E -xc - /dev/null  |
> grep
> > > LOCK
> > > > #define __GCC_ATOMIC_CHAR_LOCK_FREE 2
> > > > #define __GCC_ATOMIC_CHAR32_T_LOCK_FREE 2
> > > > #define __GCC_ATOMIC_BOOL_LOCK_FREE 2
> > > > #define __GCC_ATOMIC_POINTER_LOCK_FREE 2
> > > > #define __GCC_ATOMIC_INT_LOCK_FREE 2
> > > > #define __GCC_ATOMIC_WCHAR_T_LOCK_FREE 2
> > > > #define __GCC_ATOMIC_LONG_LOCK_FREE 2
> > > > #define __GCC_ATOMIC_CHAR16_T_LOCK_FREE 2
> > > > #define __GCC_ATOMIC_LLONG_LOCK_FREE 2
> > > > #define __GCC_ATOMIC_SHORT_LOCK_FREE 2
> > >
> > > This means it's always lock-free, implemented natively thus doesn't
> > > require libatomic. Value 1 means "sometimes lock-free" and implemented
> > > as a function provided by libatomic.
> > >
> > > Did the problem appear when I changed the flags in the makefile ? Maybe
> > > I accidently lost one and it's falling back to a subset of the target
> > > arch ?
> > >
> >
> >
> > the problem appears only with QuicTLS manually built with "-m32" flag. it
> > does not appear with "-m32" if built and linked against system OpenSS:L
> >
> > but after I modify condition (the same as previously enforcing libatomic
> in
> > ADDLIB), it builds fine.
>
> OK thanks for that, but was it already present before my changes in the
> makefile ? Could you check that the -m32 flag is properly passed to this
> test ?
>
> On 32-bit ix86, there are different cases that require libatomic:
>
>   $ gcc -march=i686 -m32 -dM -E -xc - < /dev/null |grep -c LOCK.*1
>   0
>   $ gcc -march=i586 -m32 -dM -E -xc - < /dev/null |grep -c LOCK.*1
>   0
>   $ gcc -march=i486 -m32 -dM -E -xc - < /dev/null |grep -c LOCK.*1
>   1
>   $ gcc -march=i386 -m32 -dM -E -xc - < /dev/null |grep -c LOCK.*1
>   10
>
> Only i386 and i486 require it. That makes me think, maybe it's quictls
> that was built against it and adds a dependency to it. You can check
> it using objdump -p|grep NEEDED.
>
> If so that would make sense to just manually add it (or any other
> required dependencies) in ADDLIB since they're here just to satisfy
> external dependencies.
>
> Willy
>

Re: Changes in HAProxy 3.0's Makefile and build options

[Willy Tarreau]

On Sat, Apr 20, 2024 at 03:11:19PM +0200,  ??? wrote:
> ??, 20 ???. 2024 ?. ? 15:07, Willy Tarreau :
> 
> > On Sat, Apr 20, 2024 at 02:49:38PM +0200,  ??? wrote:
> > > ??, 11 ???. 2024 ?. ? 21:05, Willy Tarreau :
> > >
> > > > Hi Ilya,
> > > >
> > > > On Thu, Apr 11, 2024 at 08:27:39PM +0200,  ??? wrote:
> > > > > do you know maybe how this was supposed to work ?
> > > > > haproxy/Makefile at master · haproxy/haproxy (github.com)
> > > > > 
> > > >
> > > > That's this:
> > > >
> > > >   ifneq ($(shell $(CC) $(CFLAGS) -dM -E -xc - /dev/null |
> > > > grep -c 'LOCK_FREE.*1'),0)
> > > > USE_LIBATOMIC   = implicit
> > > >   endif
> > > >
> > > > It calls the compiler with the known flags and checks if for this arch,
> > > > it's configured to require libatomic.
> > > >
> > >
> > > macros has changed from 1 to 2:
> > >
> > > ilia@fedora:~/Downloads$ cc -dM -E -xc - /dev/null  | grep
> > LOCK
> > > #define __GCC_ATOMIC_CHAR_LOCK_FREE 2
> > > #define __GCC_ATOMIC_CHAR32_T_LOCK_FREE 2
> > > #define __GCC_ATOMIC_BOOL_LOCK_FREE 2
> > > #define __GCC_ATOMIC_POINTER_LOCK_FREE 2
> > > #define __GCC_ATOMIC_INT_LOCK_FREE 2
> > > #define __GCC_ATOMIC_WCHAR_T_LOCK_FREE 2
> > > #define __GCC_ATOMIC_LONG_LOCK_FREE 2
> > > #define __GCC_ATOMIC_CHAR16_T_LOCK_FREE 2
> > > #define __GCC_ATOMIC_LLONG_LOCK_FREE 2
> > > #define __GCC_ATOMIC_SHORT_LOCK_FREE 2
> >
> > This means it's always lock-free, implemented natively thus doesn't
> > require libatomic. Value 1 means "sometimes lock-free" and implemented
> > as a function provided by libatomic.
> >
> > Did the problem appear when I changed the flags in the makefile ? Maybe
> > I accidently lost one and it's falling back to a subset of the target
> > arch ?
> >
> 
> 
> the problem appears only with QuicTLS manually built with "-m32" flag. it
> does not appear with "-m32" if built and linked against system OpenSS:L
> 
> but after I modify condition (the same as previously enforcing libatomic in
> ADDLIB), it builds fine.

OK thanks for that, but was it already present before my changes in the
makefile ? Could you check that the -m32 flag is properly passed to this
test ?

On 32-bit ix86, there are different cases that require libatomic:

  $ gcc -march=i686 -m32 -dM -E -xc - < /dev/null |grep -c LOCK.*1
  0
  $ gcc -march=i586 -m32 -dM -E -xc - < /dev/null |grep -c LOCK.*1
  0
  $ gcc -march=i486 -m32 -dM -E -xc - < /dev/null |grep -c LOCK.*1
  1
  $ gcc -march=i386 -m32 -dM -E -xc - < /dev/null |grep -c LOCK.*1
  10

Only i386 and i486 require it. That makes me think, maybe it's quictls
that was built against it and adds a dependency to it. You can check
it using objdump -p|grep NEEDED.

If so that would make sense to just manually add it (or any other
required dependencies) in ADDLIB since they're here just to satisfy
external dependencies.

Willy

Re: Changes in HAProxy 3.0's Makefile and build options

[Илья Шипицин]

сб, 20 апр. 2024 г. в 15:07, Willy Tarreau :

> On Sat, Apr 20, 2024 at 02:49:38PM +0200,  ??? wrote:
> > ??, 11 ???. 2024 ?. ? 21:05, Willy Tarreau :
> >
> > > Hi Ilya,
> > >
> > > On Thu, Apr 11, 2024 at 08:27:39PM +0200,  ??? wrote:
> > > > do you know maybe how this was supposed to work ?
> > > > haproxy/Makefile at master · haproxy/haproxy (github.com)
> > > > 
> > >
> > > That's this:
> > >
> > >   ifneq ($(shell $(CC) $(CFLAGS) -dM -E -xc - /dev/null |
> > > grep -c 'LOCK_FREE.*1'),0)
> > > USE_LIBATOMIC   = implicit
> > >   endif
> > >
> > > It calls the compiler with the known flags and checks if for this arch,
> > > it's configured to require libatomic.
> > >
> >
> > macros has changed from 1 to 2:
> >
> > ilia@fedora:~/Downloads$ cc -dM -E -xc - /dev/null  | grep
> LOCK
> > #define __GCC_ATOMIC_CHAR_LOCK_FREE 2
> > #define __GCC_ATOMIC_CHAR32_T_LOCK_FREE 2
> > #define __GCC_ATOMIC_BOOL_LOCK_FREE 2
> > #define __GCC_ATOMIC_POINTER_LOCK_FREE 2
> > #define __GCC_ATOMIC_INT_LOCK_FREE 2
> > #define __GCC_ATOMIC_WCHAR_T_LOCK_FREE 2
> > #define __GCC_ATOMIC_LONG_LOCK_FREE 2
> > #define __GCC_ATOMIC_CHAR16_T_LOCK_FREE 2
> > #define __GCC_ATOMIC_LLONG_LOCK_FREE 2
> > #define __GCC_ATOMIC_SHORT_LOCK_FREE 2
>
> This means it's always lock-free, implemented natively thus doesn't
> require libatomic. Value 1 means "sometimes lock-free" and implemented
> as a function provided by libatomic.
>
> Did the problem appear when I changed the flags in the makefile ? Maybe
> I accidently lost one and it's falling back to a subset of the target
> arch ?
>


the problem appears only with QuicTLS manually built with "-m32" flag. it
does not appear with "-m32" if built and linked against system OpenSS:L

but after I modify condition (the same as previously enforcing libatomic in
ADDLIB), it builds fine.


>
> > the following patch works, but I'll play a bit more 
> >
> > diff --git a/Makefile b/Makefile
> > index 4bd263498..370ac7ed0 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -493,7 +493,7 @@ $(set_target_defaults)
> >  # linking with it by default as it's not always available nor deployed
> >  # (especially on archs which do not need it).
> >  ifneq ($(USE_THREAD:0=),)
> > -  ifneq ($(shell $(CC) $(OPT_CFLAGS) $(ARCH_FLAGS) $(CPU_CFLAGS)
> > $(STD_CFLAGS) $(WARN_CFLAGS) $(NOWARN_CFLAGS) $(ERROR_CFLAGS) $(CFLAGS)
> -dM
> > -E -xc - /dev/null | grep -c 'LOCK_FREE.*1'),0)
> > +  ifneq ($(shell $(CC) $(OPT_CFLAGS) $(ARCH_FLAGS) $(CPU_CFLAGS)
> > $(STD_CFLAGS) $(WARN_CFLAGS) $(NOWARN_CFLAGS) $(ERROR_CFLAGS) $(CFLAGS)
> -dM
> > -E -xc - /dev/null | grep -c 'LOCK_FREE.*[12]'),0)
> >  USE_LIBATOMIC   = implicit
> >endif
> >  endif
>
> It would impose libatomic for everyone, and not everyone has it. For
> example this would break clang on freebsd from what I'm seeing. That's
> not logic, there must be another reason.
>
> Willy
>

Re: Changes in HAProxy 3.0's Makefile and build options

[Willy Tarreau]

On Sat, Apr 20, 2024 at 02:49:38PM +0200,  ??? wrote:
> ??, 11 ???. 2024 ?. ? 21:05, Willy Tarreau :
> 
> > Hi Ilya,
> >
> > On Thu, Apr 11, 2024 at 08:27:39PM +0200,  ??? wrote:
> > > do you know maybe how this was supposed to work ?
> > > haproxy/Makefile at master · haproxy/haproxy (github.com)
> > > 
> >
> > That's this:
> >
> >   ifneq ($(shell $(CC) $(CFLAGS) -dM -E -xc - /dev/null |
> > grep -c 'LOCK_FREE.*1'),0)
> > USE_LIBATOMIC   = implicit
> >   endif
> >
> > It calls the compiler with the known flags and checks if for this arch,
> > it's configured to require libatomic.
> >
> 
> macros has changed from 1 to 2:
> 
> ilia@fedora:~/Downloads$ cc -dM -E -xc - /dev/null  | grep LOCK
> #define __GCC_ATOMIC_CHAR_LOCK_FREE 2
> #define __GCC_ATOMIC_CHAR32_T_LOCK_FREE 2
> #define __GCC_ATOMIC_BOOL_LOCK_FREE 2
> #define __GCC_ATOMIC_POINTER_LOCK_FREE 2
> #define __GCC_ATOMIC_INT_LOCK_FREE 2
> #define __GCC_ATOMIC_WCHAR_T_LOCK_FREE 2
> #define __GCC_ATOMIC_LONG_LOCK_FREE 2
> #define __GCC_ATOMIC_CHAR16_T_LOCK_FREE 2
> #define __GCC_ATOMIC_LLONG_LOCK_FREE 2
> #define __GCC_ATOMIC_SHORT_LOCK_FREE 2

This means it's always lock-free, implemented natively thus doesn't
require libatomic. Value 1 means "sometimes lock-free" and implemented
as a function provided by libatomic.

Did the problem appear when I changed the flags in the makefile ? Maybe
I accidently lost one and it's falling back to a subset of the target
arch ?

> the following patch works, but I'll play a bit more 
> 
> diff --git a/Makefile b/Makefile
> index 4bd263498..370ac7ed0 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -493,7 +493,7 @@ $(set_target_defaults)
>  # linking with it by default as it's not always available nor deployed
>  # (especially on archs which do not need it).
>  ifneq ($(USE_THREAD:0=),)
> -  ifneq ($(shell $(CC) $(OPT_CFLAGS) $(ARCH_FLAGS) $(CPU_CFLAGS)
> $(STD_CFLAGS) $(WARN_CFLAGS) $(NOWARN_CFLAGS) $(ERROR_CFLAGS) $(CFLAGS) -dM
> -E -xc - /dev/null | grep -c 'LOCK_FREE.*1'),0)
> +  ifneq ($(shell $(CC) $(OPT_CFLAGS) $(ARCH_FLAGS) $(CPU_CFLAGS)
> $(STD_CFLAGS) $(WARN_CFLAGS) $(NOWARN_CFLAGS) $(ERROR_CFLAGS) $(CFLAGS) -dM
> -E -xc - /dev/null | grep -c 'LOCK_FREE.*[12]'),0)
>  USE_LIBATOMIC   = implicit
>endif
>  endif

It would impose libatomic for everyone, and not everyone has it. For
example this would break clang on freebsd from what I'm seeing. That's
not logic, there must be another reason.

Willy

Re: Changes in HAProxy 3.0's Makefile and build options

[Илья Шипицин]

чт, 11 апр. 2024 г. в 21:05, Willy Tarreau :

> Hi Ilya,
>
> On Thu, Apr 11, 2024 at 08:27:39PM +0200,  ??? wrote:
> > do you know maybe how this was supposed to work ?
> > haproxy/Makefile at master · haproxy/haproxy (github.com)
> > 
>
> That's this:
>
>   ifneq ($(shell $(CC) $(CFLAGS) -dM -E -xc - /dev/null |
> grep -c 'LOCK_FREE.*1'),0)
> USE_LIBATOMIC   = implicit
>   endif
>
> It calls the compiler with the known flags and checks if for this arch,
> it's configured to require libatomic.
>

macros has changed from 1 to 2:

ilia@fedora:~/Downloads$ cc -dM -E -xc - /dev/null  | grep LOCK
#define __GCC_ATOMIC_CHAR_LOCK_FREE 2
#define __GCC_ATOMIC_CHAR32_T_LOCK_FREE 2
#define __GCC_ATOMIC_BOOL_LOCK_FREE 2
#define __GCC_ATOMIC_POINTER_LOCK_FREE 2
#define __GCC_ATOMIC_INT_LOCK_FREE 2
#define __GCC_ATOMIC_WCHAR_T_LOCK_FREE 2
#define __GCC_ATOMIC_LONG_LOCK_FREE 2
#define __GCC_ATOMIC_CHAR16_T_LOCK_FREE 2
#define __GCC_ATOMIC_LLONG_LOCK_FREE 2
#define __GCC_ATOMIC_SHORT_LOCK_FREE 2

the following patch works, but I'll play a bit more 

diff --git a/Makefile b/Makefile
index 4bd263498..370ac7ed0 100644
--- a/Makefile
+++ b/Makefile
@@ -493,7 +493,7 @@ $(set_target_defaults)
 # linking with it by default as it's not always available nor deployed
 # (especially on archs which do not need it).
 ifneq ($(USE_THREAD:0=),)
-  ifneq ($(shell $(CC) $(OPT_CFLAGS) $(ARCH_FLAGS) $(CPU_CFLAGS)
$(STD_CFLAGS) $(WARN_CFLAGS) $(NOWARN_CFLAGS) $(ERROR_CFLAGS) $(CFLAGS) -dM
-E -xc - /dev/null | grep -c 'LOCK_FREE.*1'),0)
+  ifneq ($(shell $(CC) $(OPT_CFLAGS) $(ARCH_FLAGS) $(CPU_CFLAGS)
$(STD_CFLAGS) $(WARN_CFLAGS) $(NOWARN_CFLAGS) $(ERROR_CFLAGS) $(CFLAGS) -dM
-E -xc - /dev/null | grep -c 'LOCK_FREE.*[12]'),0)
 USE_LIBATOMIC   = implicit
   endif
 endif



>
> > I had to enable atomic explicitly despite it was supposed to be detected
> on
> > the fly (I haven't had a deeper look yet)
> >
> > haproxy/.github/workflows/fedora-rawhide.yml at master · haproxy/haproxy
> > <
> https://github.com/haproxy/haproxy/blob/master/.github/workflows/fedora-rawhide.yml#L17-L18
> >
>
> I honestly don't know why it's not working, it could be useful to achive
> the output of this command, either by calling it directly or for example
> by inserting "tee /tmp/log |" before grep. If you have a copy of the linker
> erorr you got without the lib, maybe it will reveal something. But honestly
> this is a perfect example of the reasons why I prefer generic makefiles to
> automatic detection: you see that you need libatomic, you add it, done. No
> need to patch a configure to mask an error due to an unhandled special case
> etc. The test above was mostly meant to ease the build for the most common
> cases (and for me till now it has always worked, on various systems and
> hardware), but I wouldn't mind too much.
>
> In fact you could simplify your build script by just passing
> USE_LIBATOMIC=1 to enable it.
>
> BTW, is there a reason for "make -j3" in the build script ? Limiting
> oneself
> to 3 build processes when modern machines rarely have less than 8 cores is
> a bit of a waste of time, especially if every other package does the same
> in the distro! I'd just do "make -j$(nproc)" as usual there.
>
> Cheers,
> Willy
>

Re: Changes in HAProxy 3.0's Makefile and build options

[Илья Шипицин]

сб, 13 апр. 2024 г. в 15:26, Willy Tarreau :

> Hi Tristan,
>
> On Fri, Apr 12, 2024 at 07:38:18AM +, Tristan wrote:
> > Hi Willy,
> >
> > > On 11 Apr 2024, at 18:18, Willy Tarreau  wrote:
> > >
> > > Some distros simply found that stuffing their regular CFLAGS into
> > > DEBUG_CFLAGS or CPU_CFLAGS does the trick most of the time. Others use
> > > other combinations depending on the tricks they figured.
> >
> > Good to know I wasn't alone scratching my head about what came from
> where!
>
> Definitely.
>
> > > After this analysis, I figured that we needed to simplify all this, get
> > > back to what is more natural to use or more commonly found, without
> > > breaking everything for everyone. [...]
> >
> > These are very nice improvements indeed, but I admit that if (at least
> > partially) breaking backwards compatibility was the acceptable choice
> here,
>
> It should almost not break it otherwise it will indicate what changed
> and how to proceed.
>
> > I'd have hoped to see something like cmake rather than a makefile
> > refactoring.
>

I would like to hear real pain behind the idea of moving to cmake.


I suspect such pain exists, for example I had to explicitly specify LUA
options depending on linux distro (it was hard to autodetect
LUA by Makefile, but it is easier for cmake)

maybe we can deal with the pain and improve things using make :)



>
> That's orthogonal. cmake is an alternative to autoconf, not to make,
> you still run "make -j$(nproc)" once cmake is done.
>
> And such mechanisms are really really a pain to deal with, at every
> stage (development and user). They can be justified for ultra-complex
> projects but quite frankly, having to imagine not being able to flip
> an option without rebuilding everything, not having something as simple
> as "V=1" to re-run the failed file and see exactly what was tried,
> having to fight against the config generator all the time etc is really
> a no-go for me. I even remember having stopped using OpenCV long ago
> when it switched from autoconf to cmake because it turned something
> complicated to something out of control that would no longer ever build
> outside of the authors' environment. We could say whatever, like they
> did it wrong or anything else, but the fact is I'm not going to impose
> a similar pain to our users.
>
> In our case, we only need to set a list of files to be built, and pass
> a few -I/-L/-l while leaving the final choice to the user.
>
> > Actually, I'd thought a few times in the past about starting a
> discussion in
> > that direction but figured it would be inconceivable.
> >
> > I don't know how controversial it is, so the main two reasons I mention
> it are:
> > - generally better tooling (and IDE support) out of the box:
> options/flags
> >   discovery and override specifically tends to be quite a bit simpler as
> the
> >   boilerplate and conventions are mostly handled by default
> > - easier to parse final result: both of them look frankly awful, but the
> >   result of cmake setups is often a little easier to parse as it
> encourages a
> >   rather declarative style (can be done in gmake, but it is very much up
> to the
> >   maintainers to be extremely disciplined about it)
>
> The problem with tools like cmake is not to parse the output when it
> works but to figure how to force it to accept that *you* are the one who
> knows when it decides it will not do what you want. While a makefile can
> trivially be overridden or even fixed, good luck for guessing all the
> magic names of cmake variables that are missing and that may help you.
>
> I really do understand why some super complex projects involving git
> submodules and stuff like this would go in that direction, but otherwise
> it's just asking for trouble with little to no benefit.
>
> > Arguably, there's the downside of requiring an extra tool everyone might
> not
> > be deeply familiar with already, and cmake versions matter more than
> gmake
> > ones so I would worry about compatibility for distros of the GCC 4 era,
> but
> > it seemed to me like it's reasonably proven and spread by now to
> consider.
>
> It's not even a matter of compatibility with any gcc version, rather a
> compatibility with what developers are able to write and what users want
> to do if that was not initially imagined by the developers. Have you read
> already about all the horrors faced by users trying to use distcc or ccache
> with cmake, often having to run sed over their cmake files ? Some time ago,
> cmake even implemented yet another magic variable specifically to make this
> less painful. And cross-compilation is yet another entire topic. All of
> this just ends up in a situation where the build system becomes an entire
> project on its own, just for the sake of passing 6 variables to make in
> the end :-/  In the case of a regular makefile at least, 100% of the
> variables you may have to use are present in the makefile, you don't need
> to resort to random guesses by 

Re: Changes in HAProxy 3.0's Makefile and build options

[Willy Tarreau]

Hi Tristan,

On Fri, Apr 12, 2024 at 07:38:18AM +, Tristan wrote:
> Hi Willy,
> 
> > On 11 Apr 2024, at 18:18, Willy Tarreau  wrote:
> > 
> > Some distros simply found that stuffing their regular CFLAGS into
> > DEBUG_CFLAGS or CPU_CFLAGS does the trick most of the time. Others use
> > other combinations depending on the tricks they figured.
> 
> Good to know I wasn't alone scratching my head about what came from where!

Definitely.

> > After this analysis, I figured that we needed to simplify all this, get
> > back to what is more natural to use or more commonly found, without
> > breaking everything for everyone. [...]
> 
> These are very nice improvements indeed, but I admit that if (at least
> partially) breaking backwards compatibility was the acceptable choice here,

It should almost not break it otherwise it will indicate what changed
and how to proceed.

> I'd have hoped to see something like cmake rather than a makefile
> refactoring.

That's orthogonal. cmake is an alternative to autoconf, not to make,
you still run "make -j$(nproc)" once cmake is done.

And such mechanisms are really really a pain to deal with, at every
stage (development and user). They can be justified for ultra-complex
projects but quite frankly, having to imagine not being able to flip
an option without rebuilding everything, not having something as simple
as "V=1" to re-run the failed file and see exactly what was tried,
having to fight against the config generator all the time etc is really
a no-go for me. I even remember having stopped using OpenCV long ago
when it switched from autoconf to cmake because it turned something
complicated to something out of control that would no longer ever build
outside of the authors' environment. We could say whatever, like they
did it wrong or anything else, but the fact is I'm not going to impose
a similar pain to our users.

In our case, we only need to set a list of files to be built, and pass
a few -I/-L/-l while leaving the final choice to the user.

> Actually, I'd thought a few times in the past about starting a discussion in
> that direction but figured it would be inconceivable.
> 
> I don't know how controversial it is, so the main two reasons I mention it 
> are:
> - generally better tooling (and IDE support) out of the box: options/flags
>   discovery and override specifically tends to be quite a bit simpler as the
>   boilerplate and conventions are mostly handled by default
> - easier to parse final result: both of them look frankly awful, but the
>   result of cmake setups is often a little easier to parse as it encourages a
>   rather declarative style (can be done in gmake, but it is very much up to 
> the
>   maintainers to be extremely disciplined about it)

The problem with tools like cmake is not to parse the output when it
works but to figure how to force it to accept that *you* are the one who
knows when it decides it will not do what you want. While a makefile can
trivially be overridden or even fixed, good luck for guessing all the
magic names of cmake variables that are missing and that may help you.

I really do understand why some super complex projects involving git
submodules and stuff like this would go in that direction, but otherwise
it's just asking for trouble with little to no benefit.

> Arguably, there's the downside of requiring an extra tool everyone might not
> be deeply familiar with already, and cmake versions matter more than gmake
> ones so I would worry about compatibility for distros of the GCC 4 era, but
> it seemed to me like it's reasonably proven and spread by now to consider.

It's not even a matter of compatibility with any gcc version, rather a
compatibility with what developers are able to write and what users want
to do if that was not initially imagined by the developers. Have you read
already about all the horrors faced by users trying to use distcc or ccache
with cmake, often having to run sed over their cmake files ? Some time ago,
cmake even implemented yet another magic variable specifically to make this
less painful. And cross-compilation is yet another entire topic. All of
this just ends up in a situation where the build system becomes an entire
project on its own, just for the sake of passing 6 variables to make in
the end :-/  In the case of a regular makefile at least, 100% of the
variables you may have to use are present in the makefile, you don't need
to resort to random guesses by copy-pasting from stackoverflow and see if
they improve your experience.

Long ago I thought that we could possibly support a config file to carry
options between versions, a bit like the .config in the Linux kernel.
Because one of the difficulties that those using autoconf/cmake is to
figure in what form to store a working setup. In practice there's no
magic solution and you end up saving the whole command line, just like
we're doing right now.

And I thought that may be some users (particularly those pulling the
dev version often) 

Re: Changes in HAProxy 3.0's Makefile and build options

[Tristan]

Hi Willy,

> On 11 Apr 2024, at 18:18, Willy Tarreau  wrote:
> 
> Some distros simply found that stuffing their regular CFLAGS into
> DEBUG_CFLAGS or CPU_CFLAGS does the trick most of the time. Others use
> other combinations depending on the tricks they figured.

Good to know I wasn’t alone scratching my head about what came from where!

> After this analysis, I figured that we needed to simplify all this, get
> back to what is more natural to use or more commonly found, without
> breaking everything for everyone. […]

These are very nice improvements indeed, but I admit that if (at least 
partially) breaking backwards compatibility was the acceptable choice here, I’d 
have hoped to see something like cmake rather than a makefile refactoring.

Actually, I’d thought a few times in the past about starting a discussion in 
that direction but figured it would be inconceivable.

I don’t know how controversial it is, so the main two reasons I mention it are:
- generally better tooling (and IDE support) out of the box: options/flags 
discovery and override specifically tends to be quite a bit simpler as the 
boilerplate and conventions are mostly handled by default
- easier to parse final result: both of them look frankly awful, but the result 
of cmake setups is often a little easier to parse as it encourages a rather 
declarative style (can be done in gmake, but it is very much up to the 
maintainers to be extremely disciplined about it)

Arguably, there’s the downside of requiring an extra tool everyone might not be 
deeply familiar with already, and cmake versions matter more than gmake ones so 
I would worry about compatibility for distros of the GCC 4 era, but it seemed 
to me like it’s reasonably proven and spread by now to consider.

Tristan

Re: Changes in HAProxy 3.0's Makefile and build options

[William Lallemand]

On Thu, Apr 11, 2024 at 11:43:14PM +0200, Dinko Korunic wrote:
> Subject: Re: Changes in HAProxy 3.0's Makefile and build options
> 
> > On 11.04.2024., at 21:32, William Lallemand  wrote:
> > 
> > If I remember correctly github actions VMs only had 2 vCPU in the past,
> > I think they upgraded to 4 vCPU last year but I can't find anything in
> > their documentation.
> 
> Hi William,
> 
> GitHub runners Instance sizes for public repositories are now as you said, 4 
> vCPU / 16 GB RAM:
> 
> https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners
> 
> 
> Kind regards,
> D.
> 

Indeed, thanks for confirming! I didn't know there was a difference with
private repositories which are still using 2 vCPU, that explains a lot.

-- 
William Lallemand

Re: Changes in HAProxy 3.0's Makefile and build options

[Dinko Korunic]

> On 11.04.2024., at 21:32, William Lallemand  wrote:
> 
> If I remember correctly github actions VMs only had 2 vCPU in the past,
> I think they upgraded to 4 vCPU last year but I can't find anything in
> their documentation.

Hi William,

GitHub runners Instance sizes for public repositories are now as you said, 4 
vCPU / 16 GB RAM:

https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners


Kind regards,
D.

-- 
Dinko Korunic   ** Standard disclaimer applies **
Sent from OSF1 osf1v4b V4.0 564 alpha

Re: Changes in HAProxy 3.0's Makefile and build options

[William Lallemand]

On Thu, Apr 11, 2024 at 09:04:51PM +0200, Willy Tarreau wrote:
> Subject: Re: Changes in HAProxy 3.0's Makefile and build options
> Hi Ilya,
> 
> On Thu, Apr 11, 2024 at 08:27:39PM +0200,  ??? wrote:
> > do you know maybe how this was supposed to work ?
> > haproxy/Makefile at master · haproxy/haproxy (github.com)
> > <https://github.com/haproxy/haproxy/blob/master/Makefile#L499>
> 
> That's this:
> 
>   ifneq ($(shell $(CC) $(CFLAGS) -dM -E -xc - /dev/null | grep 
> -c 'LOCK_FREE.*1'),0)
> USE_LIBATOMIC   = implicit
>   endif
> 
> It calls the compiler with the known flags and checks if for this arch,
> it's configured to require libatomic.
> 
> > I had to enable atomic explicitly despite it was supposed to be detected on
> > the fly (I haven't had a deeper look yet)
> > 
> > haproxy/.github/workflows/fedora-rawhide.yml at master · haproxy/haproxy
> > <https://github.com/haproxy/haproxy/blob/master/.github/workflows/fedora-rawhide.yml#L17-L18>
> 
> I honestly don't know why it's not working, it could be useful to achive
> the output of this command, either by calling it directly or for example
> by inserting "tee /tmp/log |" before grep. If you have a copy of the linker
> erorr you got without the lib, maybe it will reveal something. But honestly
> this is a perfect example of the reasons why I prefer generic makefiles to
> automatic detection: you see that you need libatomic, you add it, done. No
> need to patch a configure to mask an error due to an unhandled special case
> etc. The test above was mostly meant to ease the build for the most common
> cases (and for me till now it has always worked, on various systems and
> hardware), but I wouldn't mind too much.
> 
> In fact you could simplify your build script by just passing
> USE_LIBATOMIC=1 to enable it.
> 
> BTW, is there a reason for "make -j3" in the build script ? Limiting oneself
> to 3 build processes when modern machines rarely have less than 8 cores is
> a bit of a waste of time, especially if every other package does the same
> in the distro! I'd just do "make -j$(nproc)" as usual there.
> 

If I remember correctly github actions VMs only had 2 vCPU in the past,
I think they upgraded to 4 vCPU last year but I can't find anything in
their documentation.

-- 
William Lallemand

Re: Changes in HAProxy 3.0's Makefile and build options

[Willy Tarreau]

Hi Ilya,

On Thu, Apr 11, 2024 at 08:27:39PM +0200,  ??? wrote:
> do you know maybe how this was supposed to work ?
> haproxy/Makefile at master · haproxy/haproxy (github.com)
> 

That's this:

  ifneq ($(shell $(CC) $(CFLAGS) -dM -E -xc - /dev/null | grep -c 
'LOCK_FREE.*1'),0)
USE_LIBATOMIC   = implicit
  endif

It calls the compiler with the known flags and checks if for this arch,
it's configured to require libatomic.

> I had to enable atomic explicitly despite it was supposed to be detected on
> the fly (I haven't had a deeper look yet)
> 
> haproxy/.github/workflows/fedora-rawhide.yml at master · haproxy/haproxy
> 

I honestly don't know why it's not working, it could be useful to achive
the output of this command, either by calling it directly or for example
by inserting "tee /tmp/log |" before grep. If you have a copy of the linker
erorr you got without the lib, maybe it will reveal something. But honestly
this is a perfect example of the reasons why I prefer generic makefiles to
automatic detection: you see that you need libatomic, you add it, done. No
need to patch a configure to mask an error due to an unhandled special case
etc. The test above was mostly meant to ease the build for the most common
cases (and for me till now it has always worked, on various systems and
hardware), but I wouldn't mind too much.

In fact you could simplify your build script by just passing
USE_LIBATOMIC=1 to enable it.

BTW, is there a reason for "make -j3" in the build script ? Limiting oneself
to 3 build processes when modern machines rarely have less than 8 cores is
a bit of a waste of time, especially if every other package does the same
in the distro! I'd just do "make -j$(nproc)" as usual there.

Cheers,
Willy

Re: Changes in HAProxy 3.0's Makefile and build options

[Илья Шипицин]

чт, 11 апр. 2024 г. в 19:18, Willy Tarreau :

> Hi all,
>
> after all the time where we've all been complaining about the difficulty
> to adjust CFLAGS during the build, I could tackle the problem for a first
> step in the right direction.
>
> First, let's start with a bit of history to explain the situation and why
> it was bad. Originally, a trivial makefile with very simple rules and a
> single object to build (haproxy.o) had just CFLAGS set to "-O2 -g" or
> something like that, and that was perfect. It was possible to just pass a
> different CFLAGS value on the "make" command line and be done with it.
>
> Options started to pile up, but in a way that remained manageable for a
> long time (e.g. add PCRE support, later dlmalloc), so CFLAGS was still the
> only thing to override if needed. With 32/64 bit variants and so on, we
> started to feel the need to split those CFLAGS into multiple pieces for
> more flexibility. But these pieces were still aggregated into CFLAGS so
> that those used to overriding it were still happy. This situation forced
> us not to break these precious CFLAGS that some users were relying on.
>
> And that went like this for a long time, though the definition of this
> CFLAGS variable became more and more complicated by inheriting from some
> automatic options. For example, in 3.0-dev7, CFLAGS is initialized to
> "$(ARCH_FLAGS) $(CPU_CFLAGS) $(DEBUG_CFLAGS) $(SPEC_CFLAGS)", i.e. it
> concatenates 4 variables (in apparence). The 4th one (SPEC_CFLAGS) is
> already a concatenation of a fixed one (WARN_CFLAGS) and a series of
> automatically detected ones (the rest of it). ARCH_FLAGS is set from a
> a fixed list of presets depending on the ARCH variable, and CPU_CFLAGS
> is set from a list of presets depending on the CPU variable. And the
> most beautiful is that CFLAGS alone is no longer sufficient since some
> of the USE_* options append their own values behind it, and we complete
> with $(TARGET_CFLAGS) $(SMALL_OPTS) $(DEFINE).
>
> Yeah I know that's ugly. We all know it. Whenever someone asks me "how can
> I enable -fsanitize=address because I'd like to run ASAN", I respond "hmmm
> it depends what options you already use and which ones area easiest to
> hack".
>
> Some distros simply found that stuffing their regular CFLAGS into
> DEBUG_CFLAGS or CPU_CFLAGS does the trick most of the time. Others use
> other combinations depending on the tricks they figured.
>
> After this analysis, I figured that we needed to simplify all this, get
> back to what is more natural to use or more commonly found, without
> breaking everything for everyone. What is certain however in the current
> situation, is that nobody is overriding CFLAGS since it's too rich, too
> complex and too unpredictable. So it was really time to reset it.
>
> Thus here's what was done:
>   - CFLAGS is now empty by default and can be passed some build options
> that are appended at the end of the automatic flags. This means that
> -Os, -ggdb3, -Wfoobar, -Wno-foobar, -I../secret-place/include etc
> will properly be honored without having to trick anything anymore.
> Thus for package maintainers, building with CFLAGS="$DISTRO_CFLAGS"
> should just get the work done.
>
>   - LDFLAGS is now empty by default and can be passed some build options
> that are prepended to the list of linker options.
>
>   - ARCH_FLAGS now defaults to -g and is passed to both the compiler and
> the linker. It may be overridden to change the word size (-m32/-m64),
> enable alternate debugging formats (-ggdb3), enable LTO (-flto),
> ASAN (-fsanitize=address) etc. It's in fact for flags that must be
> consistent between the compiler and the linker. It was not renamed
> since it was already there and used quite a bit already.
>
>   - CPU_CFLAGS was preserved for ease of porting but is empty by default.
> Some may find it convenient to pass their -march, -mtune etc there.
>
>   - CPU and ARCH are gone. Let's face it, only 2 of them were usable and
> no single maintainer will be crazy enough to use these options here
> and resort to another approach for other archs. However the previous
> values are mentioned in the doc as a hint about what's known to work
> well.
>
>   - OPT_CFLAGS was created with "-O2" and nothing else. As developers,
> we spend our time juggling between -O2 and -O0 depending on whether
> we're testing or debugging. Some embedded distros also have options
> to pass -O2 or -Os to choose between fast and small, and that may
> fit very well there.
>
>   - STD_CFLAGS contains a few flags related to standards compliance.
> It is currently set to -fwrapv, -fno-strict-overflow and/or empty
> depending on what the compiler prefers. It's important not to touch
> it unless you know exactly what you're doing, and previously these
> options used to be lost by accident when overriding other ones.
>
>   - WARN_CFLAGS is now set to the list of warnings to 

Changes in HAProxy 3.0's Makefile and build options

[Willy Tarreau]

Hi all,

after all the time where we've all been complaining about the difficulty
to adjust CFLAGS during the build, I could tackle the problem for a first
step in the right direction.

First, let's start with a bit of history to explain the situation and why
it was bad. Originally, a trivial makefile with very simple rules and a
single object to build (haproxy.o) had just CFLAGS set to "-O2 -g" or
something like that, and that was perfect. It was possible to just pass a
different CFLAGS value on the "make" command line and be done with it.

Options started to pile up, but in a way that remained manageable for a
long time (e.g. add PCRE support, later dlmalloc), so CFLAGS was still the
only thing to override if needed. With 32/64 bit variants and so on, we
started to feel the need to split those CFLAGS into multiple pieces for
more flexibility. But these pieces were still aggregated into CFLAGS so
that those used to overriding it were still happy. This situation forced
us not to break these precious CFLAGS that some users were relying on.

And that went like this for a long time, though the definition of this
CFLAGS variable became more and more complicated by inheriting from some
automatic options. For example, in 3.0-dev7, CFLAGS is initialized to
"$(ARCH_FLAGS) $(CPU_CFLAGS) $(DEBUG_CFLAGS) $(SPEC_CFLAGS)", i.e. it
concatenates 4 variables (in apparence). The 4th one (SPEC_CFLAGS) is
already a concatenation of a fixed one (WARN_CFLAGS) and a series of
automatically detected ones (the rest of it). ARCH_FLAGS is set from a
a fixed list of presets depending on the ARCH variable, and CPU_CFLAGS
is set from a list of presets depending on the CPU variable. And the
most beautiful is that CFLAGS alone is no longer sufficient since some
of the USE_* options append their own values behind it, and we complete
with $(TARGET_CFLAGS) $(SMALL_OPTS) $(DEFINE).

Yeah I know that's ugly. We all know it. Whenever someone asks me "how can
I enable -fsanitize=address because I'd like to run ASAN", I respond "hmmm
it depends what options you already use and which ones area easiest to
hack".

Some distros simply found that stuffing their regular CFLAGS into
DEBUG_CFLAGS or CPU_CFLAGS does the trick most of the time. Others use
other combinations depending on the tricks they figured.

After this analysis, I figured that we needed to simplify all this, get
back to what is more natural to use or more commonly found, without
breaking everything for everyone. What is certain however in the current
situation, is that nobody is overriding CFLAGS since it's too rich, too
complex and too unpredictable. So it was really time to reset it.

Thus here's what was done:
  - CFLAGS is now empty by default and can be passed some build options
that are appended at the end of the automatic flags. This means that
-Os, -ggdb3, -Wfoobar, -Wno-foobar, -I../secret-place/include etc
will properly be honored without having to trick anything anymore.
Thus for package maintainers, building with CFLAGS="$DISTRO_CFLAGS"
should just get the work done.

  - LDFLAGS is now empty by default and can be passed some build options
that are prepended to the list of linker options.

  - ARCH_FLAGS now defaults to -g and is passed to both the compiler and
the linker. It may be overridden to change the word size (-m32/-m64),
enable alternate debugging formats (-ggdb3), enable LTO (-flto),
ASAN (-fsanitize=address) etc. It's in fact for flags that must be
consistent between the compiler and the linker. It was not renamed
since it was already there and used quite a bit already.

  - CPU_CFLAGS was preserved for ease of porting but is empty by default.
Some may find it convenient to pass their -march, -mtune etc there.

  - CPU and ARCH are gone. Let's face it, only 2 of them were usable and
no single maintainer will be crazy enough to use these options here
and resort to another approach for other archs. However the previous
values are mentioned in the doc as a hint about what's known to work
well.

  - OPT_CFLAGS was created with "-O2" and nothing else. As developers,
we spend our time juggling between -O2 and -O0 depending on whether
we're testing or debugging. Some embedded distros also have options
to pass -O2 or -Os to choose between fast and small, and that may
fit very well there.

  - STD_CFLAGS contains a few flags related to standards compliance.
It is currently set to -fwrapv, -fno-strict-overflow and/or empty
depending on what the compiler prefers. It's important not to touch
it unless you know exactly what you're doing, and previously these
options used to be lost by accident when overriding other ones.

  - WARN_CFLAGS is now set to the list of warnings to be enabled. It's
discovered automatically but may be overridden without breaking the
rest.

  - NOWARN_CFLAGS is now set to the list of warnings to be disabled. It's
discovered automatically byt