Re: Error While deviceatlas 3.2.2 and haproxy 2.9.6 make from source
hi and sorry for the long reply.
I will let you know once it is officially release, it needs to pass our QA
test still.
Kind regards.
On Mon, 6 May 2024 at 22:52, Mahendra Patil
wrote:
> any update when we can get 3.2.3 release
>
> On Wed, Apr 3, 2024 at 10:51 AM David CARLIER wrote:
>
>> Hi all,
>>
>> Thanks for your report. This is a known issue the 3.2.3 release is
>> scheduled within this month.
>>
>> Regards.
>>
>> On Wed, 3 Apr 2024 at 04:38, Willy Tarreau wrote:
>>
>>> Hello,
>>>
>>> On Wed, Apr 03, 2024 at 05:21:03AM +0530, Mahendra Patil wrote:
>>> > /opt/deviceatlas/Src//dac.c: In function ātoverdecā:
>>> > /opt/deviceatlas/Src//dac.c:714:13: warning: implicit declaration of
>>> > function ā__builtin_sadd_overflowā [-Wimplicit-function-declaration]
>>> > if (DATLAS_A_OFLOW(cur * 10, decrm, )) {
>>> (...)
>>> > /opt/deviceatlas/Src//dac.o: In function `toverdec':
>>> > /opt/deviceatlas/Src//dac.c:714: undefined reference to
>>> > `__builtin_sadd_overflow'
>>> > collect2: error: ld returned 1 exit status
>>> > make: *** [haproxy] Error 1
>>>
>>> From what I'm seeing, __builtin_sadd_overflow() first appeared in gcc-5,
>>> so you don't have it on your system, which seems to be RHEL 7 or CentOS 7
>>> based on the compiler version (gcc 4.8.5).
>>>
>>> I don't know how important is the use of this builtin for Device Atlas,
>>> I'll let David check. As a hack you could verify that it builds when you
>>> change it to:
>>>
>>> if ((r = cur*10 + decrm), 0) {
>>>
>>> But be careful that removing this overflow check might introduce a
>>> vulnerability, so if this builds, please do not deploy such code without
>>> David's approval.
>>>
>>> Another approach could be to build gcc-5.5 on your distro. It's not that
>>> hard but might not be what you were expecting to do. There are various
>>> howtos on the net, such as here:
>>>
>>> https://gist.github.com/tyleransom/2c96f53a828831567218eeb7edc2b1e7
>>>
>>> Though this one will replace the default compiler in your path, and you
>>> may likely want to add "--program-suffix=-5.5" to the configure (and
>>> replace 5.4 with 5.5 everywhere) so that you can then pass "CC=gcc-5.5"
>>> to haproxy's "make" command line.
>>>
>>> Hoping this helps,
>>> Willy
>>>
>>
>
Fwd: [PATCH] MEDIUM: shctx naming shared memory context
-- Forwarded message -
From: David CARLIER
Date: Wed, 24 Apr 2024 at 07:51
Subject: Re: [PATCH] MEDIUM: shctx naming shared memory context
To: Willy Tarreau
Hi
On Wed, 24 Apr 2024 at 07:45, Willy Tarreau wrote:
> Hi David,
>
> On Sat, Apr 20, 2024 at 07:33:16AM +0100, David CARLIER wrote:
> > From d49d9d5966caead320f33f789578cb69f2aa3787 Mon Sep 17 00:00:00 2001
> > From: David Carlier
> > Date: Sat, 20 Apr 2024 07:18:48 +0100
> > Subject: [PATCH] MEDIUM: shctx: Naming shared memory context
> >
> > From Linux 5.17, anonymous regions can be name via prctl/PR_SET_VMA
> > so caches can be identified when looking at HAProxy process memory
> > mapping.
>
> That's pretty cool, I wasn't aware of this feature, that's really
> interesting!
>
> However I disagree with this point:
>
> > +#if defined(USE_PRCTL) && defined(PR_SET_VMA)
> > + {
> > + /**
> > + * From Linux 5.17 (and if the `CONFIG_ANON_VMA_NAME`
> kernel config is set)`,
> > + * anonymous regions can be named.
> > + *
> > + * The naming can take up to 79 characters, accepting
> valid ASCII values
> > + * except [, ], \, $ and '.
> > + * As a result, when looking for /proc//maps, we can
> see the anonymous range
> > + * as follow :
> > + * `7364c4fff000-73650800 rw-s 00:01 3540
> [anon_shmem:HAProxy globalCache]`
> > + */
> > + int rn;
> > + char fullname[80];
> > +
> > + rn = snprintf(fullname, sizeof(fullname), "HAProxy %s",
> name);
> > + if (rn < 0 || prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME,
> (uintptr_t)shctx,
> > +totalsize, (uintptr_t)fullname) != 0) {
> > + munmap(shctx, totalsize);
> > + shctx = NULL;
> > + ret = SHCTX_E_ALLOC_CACHE;
> > + goto err;
> > + }
> > + }
> > +#endif
>
> You're making the mapping fail on any kernel that does not support the
> feature (hence apparently anything < 5.17 according to your description).
> IMHO we should just silently fall back to the default behavior,
Usually when using this feature, it is what is usually done indeed. However,
I thought with HAProxy you would prefer this behavior I guess I was wrong
:)
Ok changing it.
> i.e. we
> couldn't assign the name, fine, we continue to run without a name, thus
> something like this:
>
> rn = snprintf(fullname, sizeof(fullname), "HAProxy %s", name);
> if (rn >= 0)
> prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, (uintptr_t)shctx,
> totalsize, (uintptr_t)fullname);
>
> Could you please adjust it and verify that it's still OK for you ?
> Likewise I can check it on a 5.15 here.
>
> Thank you!
> Willy
>
Fwd: [PATCH] MEDIUM: shctx naming shared memory context
-- Forwarded message -
From: David CARLIER
Date: Wed, 24 Apr 2024 at 07:56
Subject: Re: [PATCH] MEDIUM: shctx naming shared memory context
To: Willy Tarreau
Here a new version.
Cheers.
On Wed, 24 Apr 2024 at 07:45, Willy Tarreau wrote:
> Hi David,
>
> On Sat, Apr 20, 2024 at 07:33:16AM +0100, David CARLIER wrote:
> > From d49d9d5966caead320f33f789578cb69f2aa3787 Mon Sep 17 00:00:00 2001
> > From: David Carlier
> > Date: Sat, 20 Apr 2024 07:18:48 +0100
> > Subject: [PATCH] MEDIUM: shctx: Naming shared memory context
> >
> > From Linux 5.17, anonymous regions can be name via prctl/PR_SET_VMA
> > so caches can be identified when looking at HAProxy process memory
> > mapping.
>
> That's pretty cool, I wasn't aware of this feature, that's really
> interesting!
>
> However I disagree with this point:
>
> > +#if defined(USE_PRCTL) && defined(PR_SET_VMA)
> > + {
> > + /**
> > + * From Linux 5.17 (and if the `CONFIG_ANON_VMA_NAME`
> kernel config is set)`,
> > + * anonymous regions can be named.
> > + *
> > + * The naming can take up to 79 characters, accepting
> valid ASCII values
> > + * except [, ], \, $ and '.
> > + * As a result, when looking for /proc//maps, we can
> see the anonymous range
> > + * as follow :
> > + * `7364c4fff000-73650800 rw-s 00:01 3540
> [anon_shmem:HAProxy globalCache]`
> > + */
> > + int rn;
> > + char fullname[80];
> > +
> > + rn = snprintf(fullname, sizeof(fullname), "HAProxy %s",
> name);
> > + if (rn < 0 || prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME,
> (uintptr_t)shctx,
> > +totalsize, (uintptr_t)fullname) != 0) {
> > + munmap(shctx, totalsize);
> > + shctx = NULL;
> > + ret = SHCTX_E_ALLOC_CACHE;
> > + goto err;
> > + }
> > + }
> > +#endif
>
> You're making the mapping fail on any kernel that does not support the
> feature (hence apparently anything < 5.17 according to your description).
> IMHO we should just silently fall back to the default behavior, i.e. we
> couldn't assign the name, fine, we continue to run without a name, thus
> something like this:
>
> rn = snprintf(fullname, sizeof(fullname), "HAProxy %s", name);
> if (rn >= 0)
> prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, (uintptr_t)shctx,
> totalsize, (uintptr_t)fullname);
>
> Could you please adjust it and verify that it's still OK for you ?
> Likewise I can check it on a 5.15 here.
>
> Thank you!
> Willy
>
From 72542db6b65994f1755d161fad04828f435f7134 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Sat, 20 Apr 2024 07:18:48 +0100
Subject: [PATCH] MEDIUM: shctx: Naming shared memory context
From Linux 5.17, anonymous regions can be name via prctl/PR_SET_VMA
so caches can be identified when looking at HAProxy process memory
mapping.
The most possible error is lack of kernel support, as a result
we ignore it, if the naming fails the mapping of memory context
ought to still occur.
---
include/haproxy/shctx.h | 2 +-
src/cache.c | 2 +-
src/shctx.c | 34 +++---
src/ssl_sock.c | 2 +-
4 files changed, 34 insertions(+), 6 deletions(-)
diff --git a/include/haproxy/shctx.h b/include/haproxy/shctx.h
index a57cf1567..01bb09d32 100644
--- a/include/haproxy/shctx.h
+++ b/include/haproxy/shctx.h
@@ -21,7 +21,7 @@
int shctx_init(struct shared_context **orig_shctx,
int maxblocks, int blocksize, unsigned int maxobjsz,
- int extra);
+ int extra, __maybe_unused const char *name);
struct shared_block *shctx_row_reserve_hot(struct shared_context *shctx,
struct shared_block *last, int data_len);
void shctx_row_detach(struct shared_context *shctx, struct shared_block *first);
diff --git a/src/cache.c b/src/cache.c
index 4a9992a19..25f20a782 100644
--- a/src/cache.c
+++ b/src/cache.c
@@ -2471,7 +2471,7 @@ int post_check_cache()
list_for_each_entry_safe(cache_config, back, _config, list) {
ret_shctx = shctx_init(, cache_config->maxblocks, CACHE_BLOCKSIZE,
- cache_config->maxobjsz, sizeof(struct cache));
+ cache_config->maxobjsz, sizeof(struct cache), cache_config->id);
if (ret_shctx <= 0) {
if (ret_shctx == SHCTX_E_INIT_LOCK)
diff --git a/src/shctx.c b/src/shctx.c
index be5905392..6c7ad172d 100644
--- a/src/shctx.c
++
[PATCH] MEDIUM: shctx naming shared memory context
Here a patch proposal for the caches feature for Linux only.
Cheers.
From d49d9d5966caead320f33f789578cb69f2aa3787 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Sat, 20 Apr 2024 07:18:48 +0100
Subject: [PATCH] MEDIUM: shctx: Naming shared memory context
From Linux 5.17, anonymous regions can be name via prctl/PR_SET_VMA
so caches can be identified when looking at HAProxy process memory
mapping.
---
include/haproxy/shctx.h | 2 +-
src/cache.c | 2 +-
src/shctx.c | 35 ---
src/ssl_sock.c | 2 +-
4 files changed, 35 insertions(+), 6 deletions(-)
diff --git a/include/haproxy/shctx.h b/include/haproxy/shctx.h
index a57cf1567..01bb09d32 100644
--- a/include/haproxy/shctx.h
+++ b/include/haproxy/shctx.h
@@ -21,7 +21,7 @@
int shctx_init(struct shared_context **orig_shctx,
int maxblocks, int blocksize, unsigned int maxobjsz,
- int extra);
+ int extra, __maybe_unused const char *name);
struct shared_block *shctx_row_reserve_hot(struct shared_context *shctx,
struct shared_block *last, int data_len);
void shctx_row_detach(struct shared_context *shctx, struct shared_block *first);
diff --git a/src/cache.c b/src/cache.c
index 4a9992a19..25f20a782 100644
--- a/src/cache.c
+++ b/src/cache.c
@@ -2471,7 +2471,7 @@ int post_check_cache()
list_for_each_entry_safe(cache_config, back, _config, list) {
ret_shctx = shctx_init(, cache_config->maxblocks, CACHE_BLOCKSIZE,
- cache_config->maxobjsz, sizeof(struct cache));
+ cache_config->maxobjsz, sizeof(struct cache), cache_config->id);
if (ret_shctx <= 0) {
if (ret_shctx == SHCTX_E_INIT_LOCK)
diff --git a/src/shctx.c b/src/shctx.c
index be5905392..dafc8d175 100644
--- a/src/shctx.c
+++ b/src/shctx.c
@@ -16,6 +16,9 @@
#include
#include
#include
+#if defined(USE_PRCTL)
+#include
+#endif
/*
* Reserve a new row if is null, put it in the hotlist, set the refcount to 1
@@ -269,13 +272,14 @@ int shctx_row_data_get(struct shared_context *shctx, struct shared_block *first,
* and 0 if cache is already allocated.
*/
int shctx_init(struct shared_context **orig_shctx, int maxblocks, int blocksize,
- unsigned int maxobjsz, int extra)
+ unsigned int maxobjsz, int extra, const char *name)
{
int i;
struct shared_context *shctx;
int ret;
void *cur;
int maptype = MAP_SHARED;
+ size_t totalsize = sizeof(struct shared_context) + extra + (maxblocks * (sizeof(struct shared_block) + blocksize));
if (maxblocks <= 0)
return 0;
@@ -284,14 +288,39 @@ int shctx_init(struct shared_context **orig_shctx, int maxblocks, int blocksize,
blocksize = (blocksize + sizeof(void *) - 1) & -sizeof(void *);
extra = (extra + sizeof(void *) - 1) & -sizeof(void *);
- shctx = (struct shared_context *)mmap(NULL, sizeof(struct shared_context) + extra + (maxblocks * (sizeof(struct shared_block) + blocksize)),
- PROT_READ | PROT_WRITE, maptype | MAP_ANON, -1, 0);
+ shctx = (struct shared_context *)mmap(NULL, totalsize, PROT_READ | PROT_WRITE, maptype | MAP_ANON, -1, 0);
if (!shctx || shctx == MAP_FAILED) {
shctx = NULL;
ret = SHCTX_E_ALLOC_CACHE;
goto err;
}
+#if defined(USE_PRCTL) && defined(PR_SET_VMA)
+ {
+ /**
+ * From Linux 5.17 (and if the `CONFIG_ANON_VMA_NAME` kernel config is set)`,
+ * anonymous regions can be named.
+ *
+ * The naming can take up to 79 characters, accepting valid ASCII values
+ * except [, ], \, $ and '.
+ * As a result, when looking for /proc//maps, we can see the anonymous range
+ * as follow :
+ * `7364c4fff000-73650800 rw-s 00:01 3540 [anon_shmem:HAProxy globalCache]`
+ */
+ int rn;
+ char fullname[80];
+
+ rn = snprintf(fullname, sizeof(fullname), "HAProxy %s", name);
+ if (rn < 0 || prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, (uintptr_t)shctx,
+totalsize, (uintptr_t)fullname) != 0) {
+ munmap(shctx, totalsize);
+ shctx = NULL;
+ ret = SHCTX_E_ALLOC_CACHE;
+ goto err;
+ }
+ }
+#endif
+
shctx->nbav = 0;
LIST_INIT(>avail);
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 9df175d93..31885efe6 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -5351,7 +5351,7 @@ int ssl_sock_prepare_bind_conf(struct bind_conf *bind_conf)
if (!ssl_shctx && global.tune.sslcachesize) {
alloc_ctx = shctx_init(_shctx, global.tune.sslcachesize,
sizeof(struct sh_ssl_sess_hdr) + SHSESS_BLOCK_MIN_SIZE, -1,
- sizeof(*sh_ssl_sess_tree));
+ sizeof(*sh_ssl_sess_tree), "ssl cache");
if (alloc_ctx <= 0) {
if (alloc_ctx == SHCTX_E_INIT_LOCK)
ha_alert("Unable to initialize the lock for the shared SSL session cache. You can retry using the
Re: Error While deviceatlas 3.2.2 and haproxy 2.9.6 make from source
Hi all,
Thanks for your report. This is a known issue the 3.2.3 release is
scheduled within this month.
Regards.
On Wed, 3 Apr 2024 at 04:38, Willy Tarreau wrote:
> Hello,
>
> On Wed, Apr 03, 2024 at 05:21:03AM +0530, Mahendra Patil wrote:
> > /opt/deviceatlas/Src//dac.c: In function ātoverdecā:
> > /opt/deviceatlas/Src//dac.c:714:13: warning: implicit declaration of
> > function ā__builtin_sadd_overflowā [-Wimplicit-function-declaration]
> > if (DATLAS_A_OFLOW(cur * 10, decrm, )) {
> (...)
> > /opt/deviceatlas/Src//dac.o: In function `toverdec':
> > /opt/deviceatlas/Src//dac.c:714: undefined reference to
> > `__builtin_sadd_overflow'
> > collect2: error: ld returned 1 exit status
> > make: *** [haproxy] Error 1
>
> From what I'm seeing, __builtin_sadd_overflow() first appeared in gcc-5,
> so you don't have it on your system, which seems to be RHEL 7 or CentOS 7
> based on the compiler version (gcc 4.8.5).
>
> I don't know how important is the use of this builtin for Device Atlas,
> I'll let David check. As a hack you could verify that it builds when you
> change it to:
>
> if ((r = cur*10 + decrm), 0) {
>
> But be careful that removing this overflow check might introduce a
> vulnerability, so if this builds, please do not deploy such code without
> David's approval.
>
> Another approach could be to build gcc-5.5 on your distro. It's not that
> hard but might not be what you were expecting to do. There are various
> howtos on the net, such as here:
>
> https://gist.github.com/tyleransom/2c96f53a828831567218eeb7edc2b1e7
>
> Though this one will replace the default compiler in your path, and you
> may likely want to add "--program-suffix=-5.5" to the configure (and
> replace 5.4 with 5.5 everywhere) so that you can then pass "CC=gcc-5.5"
> to haproxy's "make" command line.
>
> Hoping this helps,
> Willy
>
Quick question - for Haproxy Technologies...
Hey, My name is David Baker from Idytocle, we help companies such as yours answer incoming calls with our team of highly-skilled US-based representatives. We’ve successfully reduced answering expenses for other businesses by up to as 43%, while offering a well managed, satisfaction-raising solution their callers love. How about an Inbound Call Answering quote? Respond here with anticipated call volume on a monthly basis, and we’ll connect shortly to verify your needs. Sincerely, David Baker Senior Coordinator Manager Idytocle
Re: PATCH 3/3: BUILD/MEDIUM: deviceatlas: addon code update
> > It broke the CI on the "all features" build: > > > https://github.com/haproxy/haproxy/actions/runs/7671640626/job/20910459829 > > /usr/bin/ld: cannot find -lcurl: No such file or directory > /usr/bin/ld: cannot find -lzip: No such file or directory > > I'm surprised because CURL_LDFLAGS and LDFLAGS didn't change from the > previous > one: > > -CURL_LDFLAGS:= $(shell $(CURL_CONFIG) --libs 2>/dev/null || echo -L > /usr/local/lib -lcurl) > -LDFLAGS += $(CURL_LDFLAGS) $(PCRE2_LDFLAGS) -lz -lzip -lpthread > > +CURL_LDFLAGS:= $(shell $(CURL_CONFIG) --libs 2>/dev/null || echo -L > /usr/local/lib -lcurl) > +OPTIONS_LDFLAGS += $(CURL_LDFLAGS) -lz -lzip -lpthread > > I guess it's definitely because these libs are not in the VM, and maybe > they'll either need to be added or we need to make it possible to disable > them (particularly if they were not needed previously), but if you could > figure why it used to work previously, that would be great. > > > OK that's it, the flags were only used to build "dadwsch". > Yes indeed that s exactly it, the module does not use curl capabilities to function, its optional ans I see we do not compile the optional compilation unit for it anyway. You guessed it right. Thanks again ! > > Apparently there's no longer such a utility in the dummy directory, nor > is there anything providing a main() anymore, so we can safely get rid > of these build flags, right ? With this it looks like it fixes it: > > diff --git a/addons/deviceatlas/Makefile.inc > b/addons/deviceatlas/Makefile.inc > index 63719bf50..8b29559f2 100644 > --- a/addons/deviceatlas/Makefile.inc > +++ b/addons/deviceatlas/Makefile.inc > @@ -4,19 +4,12 @@ >CXX := c++ >CXXLIB := -lstdc++ > > -CURL_CONFIG := curl-config > -CURLDIR := $(shell $(CURL_CONFIG) --prefix 2>/dev/null || echo > /usr/local) > -CURL_INC:= $(CURLDIR)/include > -CURL_LIB:= $(CURLDIR)/lib > -CURL_LDFLAGS:= $(shell $(CURL_CONFIG) --libs 2>/dev/null || echo -L > /usr/local/lib -lcurl) > - >ifeq ($(DEVICEATLAS_SRC),) >OPTIONS_LDFLAGS += -lda >else >DEVICEATLAS_INC = $(DEVICEATLAS_SRC) >DEVICEATLAS_LIB = $(DEVICEATLAS_SRC) > -OPTIONS_LDFLAGS += $(CURL_LDFLAGS) -lz -lzip -lpthread > -OPTIONS_CFLAGS += -DDATLAS_CURL -DDATLAS_CURLSSET -DDATLAS_GZ > -DDATLAS_ZIP > +OPTIONS_LDFLAGS += -lpthread >OPTIONS_CFLAGS += -I$(DEVICEATLAS_INC) -I$(CURL_INC) >ifeq ($(DEVICEATLAS_NOCACHE),) >CXXFLAGS:= $(OPTIONS_CFLAGS) -std=gnu++11 > > $ ldd ./haproxy > linux-vdso.so.1 (0x7ffc82b2e000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x7f0863fc4000) > libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x7f0863c43000) > libcrypt.so.1 => /lib64/libcrypt.so.1 (0x7f0863a0b000) > libssl.so.1 => /lib64/libssl.so.1 (0x7f0863798000) > libcrypto.so.1 => /lib64/libcrypto.so.1 (0x7f086334) > libdl.so.2 => /lib64/libdl.so.2 (0x7f086313c000) > librt.so.1 => /lib64/librt.so.1 (0x7f0862f34000) > libm.so.6 => /lib64/libm.so.6 (0x7f0862c2b000) > libpcreposix.so.0 => /lib64/libpcreposix.so.0 (0x7f0862a28000) > libpcre.so.1 => /lib64/libpcre.so.1 (0x7f08627b6000) > libc.so.6 => /lib64/libc.so.6 (0x7f08623ed000) > /lib64/ld-linux-x86-64.so.2 (0x7f08641e1000) > libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x7f08621d6000) > > What do you think ? > > Willy >
Re: PATCH 3/3: BUILD/MEDIUM: deviceatlas: addon code update
I m good with your version. Thanks ! On Fri, Jan 26, 2024 at 5:35 PM Willy Tarreau wrote: > On Fri, Jan 26, 2024 at 04:41:36PM +0000, David Carlier wrote: > > Hi, > > > > Please find the revised patch. > > OK thanks, it looks good and addresses the build issue. > > I noticed that when building with the dummy lib, we continue to link > with -lstdc++ even if it's not used (unless DEVICEATLAS_NOCACHE=1) > while nothing was using c++. At first I thought it was useless, but > I spotted that there was this for the dummy lib: > > OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dacache.cpp > > instead of: > > OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dacache.o > > So I fixed it and it went fine. > > I'm still seeing another issue, the last patch adds a check on > $(uname -s) to figure dependencies, which should never be used in a > makefile since it breaks cross-compilation. From what I'm seeing, the > offending part is: > > OS := $(shell uname -s) > ... > ifeq ($(OS), Linux) > LDFLAGS += -lrt > endif > ifeq ($(OS), SunOS) > LDFLAGS += -lrt > endif > > We already have this in the main makefile: > > ifneq ($(USE_RT),) > RT_LDFLAGS = -lrt > endif > > And USE_RT is automatically set on Linux and Solaris targets, so normally > you should be able to build this new check. > > Another point, I'm seeing that the c++ output is printed not "prettified", > the whole c++ command line is always printed. With a small change I have > it clean like other ones: > > CXX addons/deviceatlas/dummy/dacache.o > CC addons/deviceatlas/dummy/dac.o > CC addons/deviceatlas/dummy/json.o > CC addons/deviceatlas/dummy/dasch.o > CC addons/deviceatlas/dummy/dadwarc.o > CC addons/deviceatlas/dummy/dadwcom.o > CC addons/deviceatlas/dummy/dadwcurl.o > CC src/version.o > LD haproxy > > The whole diff is below, if you're OK, I'll directly apply the changes > there instead of doing another round trip, otherwise feel free to suggest > a different approach (I'll merge the verbose.mk change separately). > > Willy > > --- > > diff --git a/addons/deviceatlas/Makefile.inc > b/addons/deviceatlas/Makefile.inc > index 6ca2be6b3..63719bf50 100644 > --- a/addons/deviceatlas/Makefile.inc > +++ b/addons/deviceatlas/Makefile.inc > @@ -1,7 +1,6 @@ > # DEVICEATLAS_SRC : DeviceAtlas API source root path > > > -OS := $(shell uname -s) > CXX := c++ > CXXLIB := -lstdc++ > > @@ -21,7 +20,7 @@ OPTIONS_CFLAGS += -DDATLAS_CURL -DDATLAS_CURLSSET > -DDATLAS_GZ -DDATLAS_ZIP > OPTIONS_CFLAGS += -I$(DEVICEATLAS_INC) -I$(CURL_INC) > ifeq ($(DEVICEATLAS_NOCACHE),) > CXXFLAGS:= $(OPTIONS_CFLAGS) -std=gnu++11 > -OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dacache.cpp > +OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dacache.o > OPTIONS_LDFLAGS += $(CXXLIB) > else > OPTIONS_CFLAGS += -DAPINOCACHE > @@ -35,9 +34,5 @@ OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dadwcurl.o > OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/Os/daunix.o > endif > > -ifeq ($(OS), Linux) > -LDFLAGS += -lrt > -endif > -ifeq ($(OS), SunOS) > -LDFLAGS += -lrt > -endif > +addons/deviceatlas/dummy/%.o:addons/deviceatlas/dummy/%.cpp > + $(cmd_CXX) $(CXXFLAGS) -c -o $@ $< > diff --git a/include/make/verbose.mk b/include/make/verbose.mk > index c37d513c4..9a546312d 100644 > --- a/include/make/verbose.mk > +++ b/include/make/verbose.mk > @@ -10,6 +10,7 @@ endif > # or to themselves depending on the verbosity level. > ifeq ($V,1) > cmd_CC = $(CC) > +cmd_CXX = $(CXX) > cmd_LD = $(LD) > cmd_AR = $(AR) > cmd_MAKE = +$(MAKE) > @@ -17,12 +18,14 @@ else > ifeq (3.81,$(firstword $(sort $(MAKE_VERSION) 3.81))) > # 3.81 or above > cmd_CC = $(info $ CC $@) $(Q)$(CC) > +cmd_CXX = $(info $ CXX $@) $(Q)$(CC) > cmd_LD = $(info $ LD $@) $(Q)$(LD) > cmd_AR = $(info $ AR $@) $(Q)$(AR) > cmd_MAKE = $(info $ MAKE$@) $(Q)+$(MAKE) > else > # 3.80 or older > cmd_CC = $(Q)echo " CC $@";$(CC) > +cmd_CXX = $(Q)echo " CXX $@";$(CC) > cmd_LD = $(Q)echo " LD $@";$(LD) > cmd_AR = $(Q)echo " AR $@";$(AR) > cmd_MAKE = $(Q)echo " MAKE$@";$(MAKE) > >
Re: PATCH 3/3: BUILD/MEDIUM: deviceatlas: addon code update
Hi, Please find the revised patch. Regards. On Fri, Jan 26, 2024 at 4:28 PM Willy Tarreau wrote: > Hi David, > > On Thu, Jan 25, 2024 at 09:26:24AM +0000, David Carlier wrote: > > Finally the last piece related to the da's dummy update and da.c changes. > > Thanks. I'm getting the following build error: > > addons/deviceatlas/da.c: In function 'da_haproxy_checkinst': > addons/deviceatlas/da.c:284:25: warning: implicit declaration of > function 'da_getdatacreationiso8601'; did you mean 'da_getdatacreation'? > [-Wimplicit-function-declaration] > addons/deviceatlas/da.c:283:31: warning: format '%s' expects argument of > type 'char *', but argument 2 has type 'int' [-Wformat=] > addons/deviceatlas/da.c: At top level: > addons/deviceatlas/da.o: In function `da_haproxy_checkinst': > .../addons/deviceatlas/da.c:283: undefined reference to > `da_getdatacreationiso8601' > collect2: error: ld returned 1 exit status > distcc[22474] ERROR: compile (null) on localhost failed > Makefile:999: recipe for target 'haproxy' failed > make: *** [haproxy] Error 1 > > I tried with this: >USE_DEVICEATLAS=1 DEVICEATLAS_SRC=addons/deviceatlas/dummy > and this: >USE_DEVICEATLAS=1 DEVICEATLAS_SRC=addons/deviceatlas/dummy > DEVICEATLAS_NOCACHE=1 > > And indeed, there's an inconsistency there: > > $ git grep da_getdatacreation > addons/deviceatlas/da.c: > da_getdatacreationiso8601(_deviceatlas.atlas)); > addons/deviceatlas/dummy/dac.c:da_getdatacreation(da_atlas_t *atlas) > addons/deviceatlas/dummy/dac.h:time_t da_getdatacreation(da_atlas_t > *atlas); > > Could you please recheck ? > > Thanks, > Willy > From 2cededb128250c51eb07757f9c789cfc5156aeab Mon Sep 17 00:00:00 2001 From: David Carlier Date: Thu, 25 Jan 2024 09:11:18 + Subject: [PATCH 3/3] BUILD/MEDIUM: deviceatlas: updating the addon part. - Reflecing the changes done in addons/deviceatlas/Makefile.inc. Enabling the cache feature and its disabling option as well. - Now the `dadwsch` application is part of the API's package for more general purposes, we remove it. - Minor and transparent to user changes into da.c's workflow, also making more noticeable some notices with appropriate logging levels. - Adding support for the new `deviceatlas-cache-size` config keyword, a no-op when the cache support is disabled. - Adding missing compilation units and relevant api updates to the dummy library version. --- Makefile | 23 +--- addons/deviceatlas/Makefile | 48 --- addons/deviceatlas/da.c | 111 ++- addons/deviceatlas/dadwsch.c | 195 --- addons/deviceatlas/dummy/Makefile| 2 +- addons/deviceatlas/dummy/dac.c | 15 +-- addons/deviceatlas/dummy/dac.h | 18 +-- addons/deviceatlas/dummy/dacache.cpp | 26 addons/deviceatlas/dummy/dadwarc.c | 1 + addons/deviceatlas/dummy/dadwcurl.c | 1 + 10 files changed, 121 insertions(+), 319 deletions(-) delete mode 100644 addons/deviceatlas/Makefile delete mode 100644 addons/deviceatlas/dadwsch.c create mode 100644 addons/deviceatlas/dummy/dacache.cpp create mode 100644 addons/deviceatlas/dummy/dadwarc.c create mode 100644 addons/deviceatlas/dummy/dadwcurl.c diff --git a/Makefile b/Makefile index 3fab62d69..df637cf68 100644 --- a/Makefile +++ b/Makefile @@ -667,28 +667,13 @@ endif ifneq ($(USE_DEVICEATLAS),) # Use DEVICEATLAS_SRC and possibly DEVICEATLAS_INC and DEVICEATLAS_LIB to force path - # to DeviceAtlas headers and libraries if needed. + # to DeviceAtlas headers and libraries if needed. In this context, DEVICEATLAS_NOCACHE + # can be used to disable the cache support if needed (this also removes the necessity of having + # a C++ toolchain installed). DEVICEATLAS_INC = $(DEVICEATLAS_SRC) DEVICEATLAS_LIB = $(DEVICEATLAS_SRC) - ifeq ($(DEVICEATLAS_SRC),) -DEVICEATLAS_LDFLAGS += -lda - else -ifeq ($(USE_PCRE),) - ifeq ($(USE_PCRE2),) -$(error the DeviceAtlas module needs the PCRE or the PCRE2 library in order to compile) - endif -endif -ifneq ($(USE_PCRE2),) - DEVICEATLAS_CFLAGS += -DDA_REGEX_HDR=\"dac_pcre2.c\" -DDA_REGEX_TAG=2 -endif -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/Os/daunix.o -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dadwcom.o -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dasch.o -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/json.o -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dac.o - endif + include addons/deviceatlas/Makefile.inc OPTIONS_OBJS += addons/deviceatlas/da.o - DEVICEATLAS_CFLAGS += $(if $(DEVICEATLAS_INC),-I$(DEVICEATLAS_INC)) $(if $(DEVICEATLAS_SRC),-DDATLAS_DA_NOCACHE) endif # Use 51DEGREES_SRC and possibly 51DEGREES_INC and 51DEGREES_LIB to force path diff --git a/addons/deviceatlas/Makefile b/addons/deviceatlas/Makefile deleted file mode 100644
PATCH 1/3: BUILD/MEDIUM: deviceatlas: build update
Hi here the first patch of the series related to the da's addon build change. From 6ee26e72c1d38a9e83c6c64e40c3ca50aa3f9ff9 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Thu, 25 Jan 2024 09:00:14 + Subject: [PATCH 1/3] BUILD/MEDIUM: deviceatlas: addon build rework. - Removing the legacy v2 support, which in turn suppress the need to set a regex engine. - Moving the options and addon into its distrinct build unit, cleaning up the main one in the process. - Adding a new option to disable the cache if desired or if having a C++ toolchain is not a possibility. --- addons/deviceatlas/Makefile.inc | 43 + 1 file changed, 43 insertions(+) create mode 100644 addons/deviceatlas/Makefile.inc diff --git a/addons/deviceatlas/Makefile.inc b/addons/deviceatlas/Makefile.inc new file mode 100644 index 0..6ca2be6b3 --- /dev/null +++ b/addons/deviceatlas/Makefile.inc @@ -0,0 +1,43 @@ +# DEVICEATLAS_SRC : DeviceAtlas API source root path + + +OS := $(shell uname -s) +CXX := c++ +CXXLIB := -lstdc++ + +CURL_CONFIG := curl-config +CURLDIR := $(shell $(CURL_CONFIG) --prefix 2>/dev/null || echo /usr/local) +CURL_INC:= $(CURLDIR)/include +CURL_LIB:= $(CURLDIR)/lib +CURL_LDFLAGS:= $(shell $(CURL_CONFIG) --libs 2>/dev/null || echo -L /usr/local/lib -lcurl) + +ifeq ($(DEVICEATLAS_SRC),) +OPTIONS_LDFLAGS += -lda +else +DEVICEATLAS_INC = $(DEVICEATLAS_SRC) +DEVICEATLAS_LIB = $(DEVICEATLAS_SRC) +OPTIONS_LDFLAGS += $(CURL_LDFLAGS) -lz -lzip -lpthread +OPTIONS_CFLAGS += -DDATLAS_CURL -DDATLAS_CURLSSET -DDATLAS_GZ -DDATLAS_ZIP +OPTIONS_CFLAGS += -I$(DEVICEATLAS_INC) -I$(CURL_INC) +ifeq ($(DEVICEATLAS_NOCACHE),) +CXXFLAGS:= $(OPTIONS_CFLAGS) -std=gnu++11 +OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dacache.cpp +OPTIONS_LDFLAGS += $(CXXLIB) +else +OPTIONS_CFLAGS += -DAPINOCACHE +endif +OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dac.o +OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/json.o +OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dasch.o +OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dadwarc.o +OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dadwcom.o +OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/dadwcurl.o +OPTIONS_OBJS+= $(DEVICEATLAS_SRC)/Os/daunix.o +endif + +ifeq ($(OS), Linux) +LDFLAGS += -lrt +endif +ifeq ($(OS), SunOS) +LDFLAGS += -lrt +endif -- 2.40.1
PATCH 2/3: DOC: deviceatlas: doc update
Here the second patch dedicated solely to the doc. From 9381e5e844aebd161299467bdfbf501c86409b6a Mon Sep 17 00:00:00 2001 From: David Carlier Date: Thu, 25 Jan 2024 09:09:07 + Subject: [PATCH 2/3] DOC: deviceatlas: update to be in line with the v3 api. Reflecting here all the changes, no longer need to cater with the legacy v2 neither. --- doc/DeviceAtlas-device-detection.txt | 32 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/doc/DeviceAtlas-device-detection.txt b/doc/DeviceAtlas-device-detection.txt index b6009180c..2f7ed9f71 100644 --- a/doc/DeviceAtlas-device-detection.txt +++ b/doc/DeviceAtlas-device-detection.txt @@ -3,15 +3,20 @@ DeviceAtlas Device Detection In order to add DeviceAtlas Device Detection support, you would need to download the API source code from https://deviceatlas.com/deviceatlas-haproxy-module. -The build supports the USE_PCRE and USE_PCRE2 options. Once extracted : +Once extracted : -$ make TARGET= USE_PCRE=1 (or USE_PCRE2=1) USE_DEVICEATLAS=1 DEVICEATLAS_SRC= +$ make TARGET= USE_DEVICEATLAS=1 DEVICEATLAS_SRC= Optionally DEVICEATLAS_INC and DEVICEATLAS_LIB may be set to override the path to the include files and libraries respectively if they're not in the source -directory. However, if the API had been installed beforehand, DEVICEATLAS_SRC -can be omitted. Note that the DeviceAtlas C API version supported is the 2.4.0 -at minimum. +directory. Also, in the case the api cache support is not needed and/or a C++ toolchain + could not be used, DEVICEATLAS_NOCACHE is available. + +$ make TARGET= USE_DEVICEATLAS=1 DEVICEATLAS_SRC= DEVICEATLAS_NOCACHE=1 + +However, if the API had been installed beforehand, DEVICEATLAS_SRC +can be omitted. Note that the DeviceAtlas C API version supported is from the 3.x +releases serie (3.2.1 minimum recommended). For HAProxy developers who need to verify that their changes didn't accidentally break the DeviceAtlas code, it is possible to build a dummy library provided in @@ -20,7 +25,7 @@ full library. This will not provide the full functionalities, it will just allow haproxy to start with a deviceatlas configuration, which generally is enough to validate API changes : -$ make TARGET= USE_PCRE=1 USE_DEVICEATLAS=1 DEVICEATLAS_SRC=$PWD/addons/deviceatlas/dummy +$ make TARGET= USE_DEVICEATLAS=1 DEVICEATLAS_SRC=$PWD/addons/deviceatlas/dummy These are supported DeviceAtlas directives (see doc/configuration.txt) : - deviceatlas-json-file . @@ -28,6 +33,7 @@ These are supported DeviceAtlas directives (see doc/configuration.txt) : the API, 0 by default). - deviceatlas-property-separator (character used to separate the properties produced by the API, | by default). + - deviceatlas-cache-size (number of cache entries, 0 by default). Sample configuration : @@ -64,18 +70,8 @@ Single HTTP header acl device_type_tablet req.fhdr(User-Agent),da-csv-conv(primaryHardwareType) "Tablet" -Optionally a JSON download scheduler is provided to allow a data file being -fetched automatically in a daily basis without restarting HAProxy : - -$ cd addons/deviceatlas && make [DEVICEATLAS_SRC=] - -Similarly, if the DeviceAtlas API is installed, DEVICEATLAS_SRC can be omitted. - -$ ./dadwsch -u JSON data file URL e.g. "https://deviceatlas.com/getJSON?licencekey==zip=my=web" \ - [-p download directory path /tmp by default] \ - [-d scheduled hour of download, hour when the service is launched by default] - -Noted it needs to be started before HAProxy. +Note that the JSON download scheduler is now part of the API's package, it is recommended +to read its documentation. Note it needs to be started before HAProxy. Please find more information about DeviceAtlas and the detection methods at -- 2.40.1
PATCH 3/3: BUILD/MEDIUM: deviceatlas: addon code update
Finally the last piece related to the da's dummy update and da.c changes. From df705a30e7d6ce41b61ac7a3e670158b2b2cac25 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Thu, 25 Jan 2024 09:11:18 + Subject: [PATCH 3/3] BUILD/MEDIUM: deviceatlas: updating the addon part. - Reflecing the changes done in addons/deviceatlas/Makefile.inc. Enabling the cache feature and its disabling option as well. - Now the `dadwsch` application is part of the API's package for more general purposes, we remove it. - Minor and transparent to user changes into da.c's workflow, also making more noticeable some notices with appropriate logging levels. - Adding support for the new `deviceatlas-cache-size` config keyword, a no-op when the cache support is disabled. - Adding missing compilation units to the dummy library version. --- Makefile | 23 +--- addons/deviceatlas/Makefile | 48 --- addons/deviceatlas/da.c | 111 ++- addons/deviceatlas/dadwsch.c | 195 --- addons/deviceatlas/dummy/Makefile| 2 +- addons/deviceatlas/dummy/dac.h | 11 +- addons/deviceatlas/dummy/dacache.cpp | 26 addons/deviceatlas/dummy/dadwarc.c | 1 + addons/deviceatlas/dummy/dadwcurl.c | 1 + 9 files changed, 112 insertions(+), 306 deletions(-) delete mode 100644 addons/deviceatlas/Makefile delete mode 100644 addons/deviceatlas/dadwsch.c create mode 100644 addons/deviceatlas/dummy/dacache.cpp create mode 100644 addons/deviceatlas/dummy/dadwarc.c create mode 100644 addons/deviceatlas/dummy/dadwcurl.c diff --git a/Makefile b/Makefile index 3fab62d69..df637cf68 100644 --- a/Makefile +++ b/Makefile @@ -667,28 +667,13 @@ endif ifneq ($(USE_DEVICEATLAS),) # Use DEVICEATLAS_SRC and possibly DEVICEATLAS_INC and DEVICEATLAS_LIB to force path - # to DeviceAtlas headers and libraries if needed. + # to DeviceAtlas headers and libraries if needed. In this context, DEVICEATLAS_NOCACHE + # can be used to disable the cache support if needed (this also removes the necessity of having + # a C++ toolchain installed). DEVICEATLAS_INC = $(DEVICEATLAS_SRC) DEVICEATLAS_LIB = $(DEVICEATLAS_SRC) - ifeq ($(DEVICEATLAS_SRC),) -DEVICEATLAS_LDFLAGS += -lda - else -ifeq ($(USE_PCRE),) - ifeq ($(USE_PCRE2),) -$(error the DeviceAtlas module needs the PCRE or the PCRE2 library in order to compile) - endif -endif -ifneq ($(USE_PCRE2),) - DEVICEATLAS_CFLAGS += -DDA_REGEX_HDR=\"dac_pcre2.c\" -DDA_REGEX_TAG=2 -endif -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/Os/daunix.o -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dadwcom.o -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dasch.o -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/json.o -OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dac.o - endif + include addons/deviceatlas/Makefile.inc OPTIONS_OBJS += addons/deviceatlas/da.o - DEVICEATLAS_CFLAGS += $(if $(DEVICEATLAS_INC),-I$(DEVICEATLAS_INC)) $(if $(DEVICEATLAS_SRC),-DDATLAS_DA_NOCACHE) endif # Use 51DEGREES_SRC and possibly 51DEGREES_INC and 51DEGREES_LIB to force path diff --git a/addons/deviceatlas/Makefile b/addons/deviceatlas/Makefile deleted file mode 100644 index fbcffca56..0 --- a/addons/deviceatlas/Makefile +++ /dev/null @@ -1,48 +0,0 @@ -# DEVICEATLAS_SRC : DeviceAtlas API source root path - - -OS := $(shell uname -s) -OBJS:= dadwsch.o -CFLAGS := -g -O2 -LDFLAGS := - -CURL_CONFIG := curl-config -CURLDIR := $(shell $(CURL_CONFIG) --prefix 2>/dev/null || echo /usr/local) -CURL_INC:= $(CURLDIR)/include -CURL_LIB:= $(CURLDIR)/lib -CURL_LDFLAGS:= $(shell $(CURL_CONFIG) --libs 2>/dev/null || echo -L /usr/local/lib -lcurl) - -PCRE2_CONFIG:= pcre2-config -PCRE2DIR:= $(shell $(PCRE2_CONFIG) --prefix 2>/dev/null || echo /usr/local) -PCRE2_INC := $(PCRE2DIR)/include -PCRE2_LIB := $(PCRE2DIR)/lib -PCRE2_LDFLAGS := $(shell $(PCRE2_CONFIG) --libs8 2>/dev/null || echo /usr/local) - -ifeq ($(DEVICEATLAS_SRC),) -dadwsch:dadwsch.c - $(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) - -LDFLAGS += -lda -else -DEVICEATLAS_INC = $(DEVICEATLAS_SRC) -DEVICEATLAS_LIB = $(DEVICEATLAS_SRC) -CFLAGS += -DDA_REGEX_HDR=\"dac_pcre2.c\" -DDA_REGEX_TAG=2 -CFLAGS += -DMOBI_CURL -DMOBI_CURLSSET -DMOBI_GZ -DMOBI_ZIP -CFLAGS += -I$(DEVICEATLAS_INC) -I$(CURL_INC) -I$(PCRE2DIR) -LDFLAGS += $(CURL_LDFLAGS) $(PCRE2_LDFLAGS) -lz -lzip -lpthread - -dadwsch:dadwsch.c $(DEVICEATLAS_SRC)/dac.c $(DEVICEATLAS_SRC)/dasch.c $(DEVICEATLAS_SRC)/dadwarc.c $(DEVICEATLAS_SRC)/dadwcom.c $(DEVICEATLAS_SRC)/dadwcurl.c $(DEVICEATLAS_SRC)/json.c $(DEVICEATLAS_SRC)/Os/daunix.c - $(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) -endif - -ifeq ($(OS), Linux) -LDFLAGS += -lrt -endif -ifeq ($(OS), SunOS) -LDFLAGS += -lrt -endif - -clean: - rm -f *.o - rm -f $(DEVICEATLAS_LIB)*.o - r
Re: Error While deviceatlas compile/make from source
Sorry for the inconvenience, there was indeed an essential missing component to be fully workable. It had been now fixed, from the same URL as before. Kindest regards. On Wed, 4 Oct 2023 at 00:08, Mahendra Patil wrote: > > Downloaded all latest and tried but still error > > [root@govinda opt]# /opt/haproxy-ssl/sbin/haproxy -f > /opt/haproxy-ssl/conf/haproxy.cfg > [NOTICE] (28500) : haproxy version is 2.8.3-86e043a > [NOTICE] (28500) : path to executable is /opt/haproxy-ssl/sbin/haproxy > > *[ALERT](28500) : config : deviceatlas : data could not be compiled.* > > > > > > On Fri, Sep 29, 2023 at 4:44 PM David CARLIER wrote: > >> Hi Mahendra, sorry for late reply but you cam download a JSON from this >> page https://deviceatlas.com/deviceatlas-haproxy-module >> now they are compatible with the V3 api. >> >> Kind regards. >> >> On Wed, 20 Sept 2023 at 23:50, Mahendra Patil >> wrote: >> >>> *After installation haproxy not able to start , show below error* >>> >>> [root@govinda opt]# /opt/haproxy-ssl/sbin/haproxy -f >>> /opt/haproxy-ssl/conf/haproxy.cfg >>> [NOTICE] (42915) : haproxy version is 2.8.3-86e043a >>> [NOTICE] (42915) : path to executable is /opt/haproxy-ssl/sbin/haproxy >>> [ALERT](42915) : config : deviceatlas : >>> '/opt/deviceatlas/20230802_compact.json' json file is invalid. >>> >>> On Tue, Sep 19, 2023 at 5:36 PM David CARLIER >>> wrote: >>> >>>> We are almost there, now remains the linkage part to pass. >>>> In your /opt/deviceatlas-enterprise-c-3.2 folder, there should be a >>>> `include` and a `lib` or a `lib64` folder >>>> So for the last command, what needs to be done instead is one of the >>>> following (no need to repeat step 1) : >>>> >>>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>>> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib >>>> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib -lda" >>>> >>>> or >>>> >>>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib64 >>>> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib64 >>>> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib64 -lda" >>>> >>>> The only difference is `lib` vs `lib64` >>>> >>>> Hope it's useful. >>>> >>>> Kindest regards. >>>> >>>> >>>> >>>> On Tue, 19 Sept 2023 at 12:52, Mahendra Patil < >>>> mahendra.pa...@naaptol.com> wrote: >>>> >>>>> >>>>> 1) >>>>> cd /opt/deviceatlas-enterprise-c-3.2 >>>>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 >>>>> -DCMAKE_C_FLAGS="-std=gnu99" >>>>> make >>>>> make install >>>>> >>>>> 2) >>>>> cd haproxy-2.8.3 >>>>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>>>> USE_DEVICEATLAS=1 >>>>> DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>>>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>>>> >>>>> LD haproxy >>>>> /usr/bin/ld: cannot find -lda >>>>> collect2: error: ld returned 1 exit status >>>>> make: *** [haproxy] Error 1 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Sep 19, 2023 at 10:19 AM David CARLIER >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Since you build the DeviceAtlas library from cmake prior, the >>>>>> following steps are more appropriate : >>>>>> 1/ Inside the deviceatlas-enterprise-c-3.2 folder >>>>>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 >>>>>> -DCMAKE_C_FLAGS="-std=gnu99" >>>>>> make >>>>>> make install >>>>>> >>>>>> 2/ Inside the haproxy-2.8.3 folder >>>>>> >>>>>> make TARGET=linux-glibc USE_
Re: Error While deviceatlas compile/make from source
Hi Mahendra, sorry for late reply but you cam download a JSON from this page https://deviceatlas.com/deviceatlas-haproxy-module now they are compatible with the V3 api. Kind regards. On Wed, 20 Sept 2023 at 23:50, Mahendra Patil wrote: > *After installation haproxy not able to start , show below error* > > [root@govinda opt]# /opt/haproxy-ssl/sbin/haproxy -f > /opt/haproxy-ssl/conf/haproxy.cfg > [NOTICE] (42915) : haproxy version is 2.8.3-86e043a > [NOTICE] (42915) : path to executable is /opt/haproxy-ssl/sbin/haproxy > [ALERT](42915) : config : deviceatlas : > '/opt/deviceatlas/20230802_compact.json' json file is invalid. > > On Tue, Sep 19, 2023 at 5:36 PM David CARLIER wrote: > >> We are almost there, now remains the linkage part to pass. >> In your /opt/deviceatlas-enterprise-c-3.2 folder, there should be a >> `include` and a `lib` or a `lib64` folder >> So for the last command, what needs to be done instead is one of the >> following (no need to repeat step 1) : >> >> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib >> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib -lda" >> >> or >> >> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib64 >> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib64 >> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib64 -lda" >> >> The only difference is `lib` vs `lib64` >> >> Hope it's useful. >> >> Kindest regards. >> >> >> >> On Tue, 19 Sept 2023 at 12:52, Mahendra Patil >> wrote: >> >>> >>> 1) >>> cd /opt/deviceatlas-enterprise-c-3.2 >>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 >>> -DCMAKE_C_FLAGS="-std=gnu99" >>> make >>> make install >>> >>> 2) >>> cd haproxy-2.8.3 >>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>> >>> LD haproxy >>> /usr/bin/ld: cannot find -lda >>> collect2: error: ld returned 1 exit status >>> make: *** [haproxy] Error 1 >>> >>> >>> >>> >>> >>> On Tue, Sep 19, 2023 at 10:19 AM David CARLIER >>> wrote: >>> >>>> Hi, >>>> >>>> Since you build the DeviceAtlas library from cmake prior, the following >>>> steps are more appropriate : >>>> 1/ Inside the deviceatlas-enterprise-c-3.2 folder >>>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 >>>> -DCMAKE_C_FLAGS="-std=gnu99" >>>> make >>>> make install >>>> >>>> 2/ Inside the haproxy-2.8.3 folder >>>> >>>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>>> >>>> Let me know if you have any further question. >>>> >>>> On Tue, 19 Sept 2023 at 00:02, Mahendra Patil < >>>> mahendra.pa...@naaptol.com> wrote: >>>> >>>>> Thanks for yours prompt reply about deviceatlas-enterprise-c-3.2 >>>>> error while make >>>>> after yours suggestion deviceatlas-enterprise-c-3.2 *make *works fine >>>>> >>>>> But while using with haproxy-2.8.3 version gives following error >>>>> >>>>> [root@govinda opt]# cd haproxy-2.8.3 >>>>> [root@govinda haproxy-2.8.3]# make TARGET=linux-glibc USE_PCRE=1 >>>>> USE_OPENSSL=1 USE_ZLIB=1 USE_DEVICEATLAS=1 >>>>> DEVICEATLAS_SRC=/opt/deviceatlas-enterprise-c-3.2/Src/ >>>>> CC src/ev_poll.o >>>>> CC src/ev_epoll.o >>>>> CC src/cpuset.o >>>>> CC src/ssl_sock.o >>>>> CC src/ssl_ckch.o >>>>> CC src/ssl_sample.o >>>>> CC src/ssl_crtlist.o >>>>> CC src/cfgparse-ssl.o >>>>> CC src/ssl_utils.o >>&g
Re: Error While deviceatlas compile/make from source
Sorry I forgot the link https://deviceatlas.com/deviceatlas-haproxy-module. On Thu, 21 Sept 2023 at 17:10, David CARLIER wrote: > So the 20230802_compact.json only works with v2 versions, even from the > page here it is still the case. > Your inquiry had been acknowledged by the dev team but I can't say exactly > when the new v3 version will be available. > But we ll keep you posted. > > Kindest regards. > > On Wed, 20 Sept 2023 at 23:50, Mahendra Patil > wrote: > >> *After installation haproxy not able to start , show below error* >> >> [root@govinda opt]# /opt/haproxy-ssl/sbin/haproxy -f >> /opt/haproxy-ssl/conf/haproxy.cfg >> [NOTICE] (42915) : haproxy version is 2.8.3-86e043a >> [NOTICE] (42915) : path to executable is /opt/haproxy-ssl/sbin/haproxy >> [ALERT](42915) : config : deviceatlas : >> '/opt/deviceatlas/20230802_compact.json' json file is invalid. >> >> On Tue, Sep 19, 2023 at 5:36 PM David CARLIER wrote: >> >>> We are almost there, now remains the linkage part to pass. >>> In your /opt/deviceatlas-enterprise-c-3.2 folder, there should be a >>> `include` and a `lib` or a `lib64` folder >>> So for the last command, what needs to be done instead is one of the >>> following (no need to repeat step 1) : >>> >>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib >>> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib -lda" >>> >>> or >>> >>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib64 >>> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib64 >>> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib64 -lda" >>> >>> The only difference is `lib` vs `lib64` >>> >>> Hope it's useful. >>> >>> Kindest regards. >>> >>> >>> >>> On Tue, 19 Sept 2023 at 12:52, Mahendra Patil < >>> mahendra.pa...@naaptol.com> wrote: >>> >>>> >>>> 1) >>>> cd /opt/deviceatlas-enterprise-c-3.2 >>>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 >>>> -DCMAKE_C_FLAGS="-std=gnu99" >>>> make >>>> make install >>>> >>>> 2) >>>> cd haproxy-2.8.3 >>>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>>> >>>> LD haproxy >>>> /usr/bin/ld: cannot find -lda >>>> collect2: error: ld returned 1 exit status >>>> make: *** [haproxy] Error 1 >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Sep 19, 2023 at 10:19 AM David CARLIER >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Since you build the DeviceAtlas library from cmake prior, the >>>>> following steps are more appropriate : >>>>> 1/ Inside the deviceatlas-enterprise-c-3.2 folder >>>>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 >>>>> -DCMAKE_C_FLAGS="-std=gnu99" >>>>> make >>>>> make install >>>>> >>>>> 2/ Inside the haproxy-2.8.3 folder >>>>> >>>>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>>>> USE_DEVICEATLAS=1 >>>>> DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>>>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>>>> >>>>> Let me know if you have any further question. >>>>> >>>>> On Tue, 19 Sept 2023 at 00:02, Mahendra Patil < >>>>> mahendra.pa...@naaptol.com> wrote: >>>>> >>>>>> Thanks for yours prompt reply about deviceatlas-enterprise-c-3.2 >>>>>> error while make >>>>>> after yours suggestion deviceatlas-enterprise-c-3.2 *make *works fine >>>>>> >>>>>> But while using with haproxy-2.8.3 version gives following error >>>>>> >>
Re: Error While deviceatlas compile/make from source
So the 20230802_compact.json only works with v2 versions, even from the page here it is still the case. Your inquiry had been acknowledged by the dev team but I can't say exactly when the new v3 version will be available. But we ll keep you posted. Kindest regards. On Wed, 20 Sept 2023 at 23:50, Mahendra Patil wrote: > *After installation haproxy not able to start , show below error* > > [root@govinda opt]# /opt/haproxy-ssl/sbin/haproxy -f > /opt/haproxy-ssl/conf/haproxy.cfg > [NOTICE] (42915) : haproxy version is 2.8.3-86e043a > [NOTICE] (42915) : path to executable is /opt/haproxy-ssl/sbin/haproxy > [ALERT](42915) : config : deviceatlas : > '/opt/deviceatlas/20230802_compact.json' json file is invalid. > > On Tue, Sep 19, 2023 at 5:36 PM David CARLIER wrote: > >> We are almost there, now remains the linkage part to pass. >> In your /opt/deviceatlas-enterprise-c-3.2 folder, there should be a >> `include` and a `lib` or a `lib64` folder >> So for the last command, what needs to be done instead is one of the >> following (no need to repeat step 1) : >> >> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib >> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib -lda" >> >> or >> >> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib64 >> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib64 >> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib64 -lda" >> >> The only difference is `lib` vs `lib64` >> >> Hope it's useful. >> >> Kindest regards. >> >> >> >> On Tue, 19 Sept 2023 at 12:52, Mahendra Patil >> wrote: >> >>> >>> 1) >>> cd /opt/deviceatlas-enterprise-c-3.2 >>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 >>> -DCMAKE_C_FLAGS="-std=gnu99" >>> make >>> make install >>> >>> 2) >>> cd haproxy-2.8.3 >>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>> >>> LD haproxy >>> /usr/bin/ld: cannot find -lda >>> collect2: error: ld returned 1 exit status >>> make: *** [haproxy] Error 1 >>> >>> >>> >>> >>> >>> On Tue, Sep 19, 2023 at 10:19 AM David CARLIER >>> wrote: >>> >>>> Hi, >>>> >>>> Since you build the DeviceAtlas library from cmake prior, the following >>>> steps are more appropriate : >>>> 1/ Inside the deviceatlas-enterprise-c-3.2 folder >>>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 >>>> -DCMAKE_C_FLAGS="-std=gnu99" >>>> make >>>> make install >>>> >>>> 2/ Inside the haproxy-2.8.3 folder >>>> >>>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>>> >>>> Let me know if you have any further question. >>>> >>>> On Tue, 19 Sept 2023 at 00:02, Mahendra Patil < >>>> mahendra.pa...@naaptol.com> wrote: >>>> >>>>> Thanks for yours prompt reply about deviceatlas-enterprise-c-3.2 >>>>> error while make >>>>> after yours suggestion deviceatlas-enterprise-c-3.2 *make *works fine >>>>> >>>>> But while using with haproxy-2.8.3 version gives following error >>>>> >>>>> [root@govinda opt]# cd haproxy-2.8.3 >>>>> [root@govinda haproxy-2.8.3]# make TARGET=linux-glibc USE_PCRE=1 >>>>> USE_OPENSSL=1 USE_ZLIB=1 USE_DEVICEATLAS=1 >>>>> DEVICEATLAS_SRC=/opt/deviceatlas-enterprise-c-3.2/Src/ >>>>> CC src/ev_poll.o >>>>> CC src/ev_epoll.o >>>>> CC src/cpuset.o >>>>> CC src/ssl_sock.o >>>>> CC src/ssl_ckch.o >>>>> CC src/ssl_sample.o >>>>> CC src/ssl_crtlist.o >>>
Re: Error While deviceatlas compile/make from source
We are almost there, now remains the linkage part to pass.
In your /opt/deviceatlas-enterprise-c-3.2 folder, there should be a
`include` and a `lib` or a `lib64` folder
So for the last command, what needs to be done instead is one of the
following (no need to repeat step 1) :
make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include
DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib
DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib
-Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib -lda"
or
make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include
DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib64
DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib64
-Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib64 -lda"
The only difference is `lib` vs `lib64`
Hope it's useful.
Kindest regards.
On Tue, 19 Sept 2023 at 12:52, Mahendra Patil
wrote:
>
> 1)
> cd /opt/deviceatlas-enterprise-c-3.2
> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2
> -DCMAKE_C_FLAGS="-std=gnu99"
> make
> make install
>
> 2)
> cd haproxy-2.8.3
> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include
> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib
>
> LD haproxy
> /usr/bin/ld: cannot find -lda
> collect2: error: ld returned 1 exit status
> make: *** [haproxy] Error 1
>
>
>
>
>
> On Tue, Sep 19, 2023 at 10:19 AM David CARLIER wrote:
>
>> Hi,
>>
>> Since you build the DeviceAtlas library from cmake prior, the following
>> steps are more appropriate :
>> 1/ Inside the deviceatlas-enterprise-c-3.2 folder
>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2
>> -DCMAKE_C_FLAGS="-std=gnu99"
>> make
>> make install
>>
>> 2/ Inside the haproxy-2.8.3 folder
>>
>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include
>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib
>>
>> Let me know if you have any further question.
>>
>> On Tue, 19 Sept 2023 at 00:02, Mahendra Patil
>> wrote:
>>
>>> Thanks for yours prompt reply about deviceatlas-enterprise-c-3.2 error
>>> while make
>>> after yours suggestion deviceatlas-enterprise-c-3.2 *make *works fine
>>>
>>> But while using with haproxy-2.8.3 version gives following error
>>>
>>> [root@govinda opt]# cd haproxy-2.8.3
>>> [root@govinda haproxy-2.8.3]# make TARGET=linux-glibc USE_PCRE=1
>>> USE_OPENSSL=1 USE_ZLIB=1 USE_DEVICEATLAS=1
>>> DEVICEATLAS_SRC=/opt/deviceatlas-enterprise-c-3.2/Src/
>>> CC src/ev_poll.o
>>> CC src/ev_epoll.o
>>> CC src/cpuset.o
>>> CC src/ssl_sock.o
>>> CC src/ssl_ckch.o
>>> CC src/ssl_sample.o
>>> CC src/ssl_crtlist.o
>>> CC src/cfgparse-ssl.o
>>> CC src/ssl_utils.o
>>> CC src/jwt.o
>>> CC src/ssl_ocsp.o
>>> CC /opt/deviceatlas/Src//Os/daunix.o
>>> CC /opt/deviceatlas/Src//dadwcom.o
>>> CC /opt/deviceatlas/Src//dasch.o
>>> CC /opt/deviceatlas/Src//json.o
>>> CC /opt/deviceatlas/Src//dac.o
>>> /opt/deviceatlas/Src//dac.c:1033:0: warning: ignoring #pragma optimize
>>> [-Wunknown-pragmas]
>>> #pragma optimize("Ofast")
>>> ^
>>> /opt/deviceatlas/Src//dac.c:1041:0: warning: ignoring #pragma optimize
>>> [-Wunknown-pragmas]
>>> #pragma optimize("Ofast")
>>> ^
>>> /opt/deviceatlas/Src//dac.c:1049:0: warning: ignoring #pragma optimize
>>> [-Wunknown-pragmas]
>>> #pragma optimize("Ofast")
>>> ^
>>> /opt/deviceatlas/Src//dac.c:1057:0: warning: ignoring #pragma optimize
>>> [-Wunknown-pragmas]
>>> #pragma optimize("Ofast")
>>> ^
>>> /opt/deviceatlas/Src//dac.c:1065:0: warning: ignoring #pragma optimize
>>> [-Wunknown-pragmas]
>>> #pragma optimize("Ofast")
>>> ^
>>> /opt/deviceatlas/Src//dac.c:1076:0: warning: ignoring #pragma optimize
>>> [-Wunknown-pragmas]
>>> #pragma optimize("Ofast")
>>> ^
>>> /opt/deviceatlas/Src//dac.c: In function ârun_san_evidenceâ:
>>> /opt/deviceatlas/Src//dac.c:1404:23: warning
Re: Error While deviceatlas compile/make from source
Hi,
Since you build the DeviceAtlas library from cmake prior, the following
steps are more appropriate :
1/ Inside the deviceatlas-enterprise-c-3.2 folder
cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2
-DCMAKE_C_FLAGS="-std=gnu99"
make
make install
2/ Inside the haproxy-2.8.3 folder
make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include
DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib
Let me know if you have any further question.
On Tue, 19 Sept 2023 at 00:02, Mahendra Patil
wrote:
> Thanks for yours prompt reply about deviceatlas-enterprise-c-3.2 error
> while make
> after yours suggestion deviceatlas-enterprise-c-3.2 *make *works fine
>
> But while using with haproxy-2.8.3 version gives following error
>
> [root@govinda opt]# cd haproxy-2.8.3
> [root@govinda haproxy-2.8.3]# make TARGET=linux-glibc USE_PCRE=1
> USE_OPENSSL=1 USE_ZLIB=1 USE_DEVICEATLAS=1
> DEVICEATLAS_SRC=/opt/deviceatlas-enterprise-c-3.2/Src/
> CC src/ev_poll.o
> CC src/ev_epoll.o
> CC src/cpuset.o
> CC src/ssl_sock.o
> CC src/ssl_ckch.o
> CC src/ssl_sample.o
> CC src/ssl_crtlist.o
> CC src/cfgparse-ssl.o
> CC src/ssl_utils.o
> CC src/jwt.o
> CC src/ssl_ocsp.o
> CC /opt/deviceatlas/Src//Os/daunix.o
> CC /opt/deviceatlas/Src//dadwcom.o
> CC /opt/deviceatlas/Src//dasch.o
> CC /opt/deviceatlas/Src//json.o
> CC /opt/deviceatlas/Src//dac.o
> /opt/deviceatlas/Src//dac.c:1033:0: warning: ignoring #pragma optimize
> [-Wunknown-pragmas]
> #pragma optimize("Ofast")
> ^
> /opt/deviceatlas/Src//dac.c:1041:0: warning: ignoring #pragma optimize
> [-Wunknown-pragmas]
> #pragma optimize("Ofast")
> ^
> /opt/deviceatlas/Src//dac.c:1049:0: warning: ignoring #pragma optimize
> [-Wunknown-pragmas]
> #pragma optimize("Ofast")
> ^
> /opt/deviceatlas/Src//dac.c:1057:0: warning: ignoring #pragma optimize
> [-Wunknown-pragmas]
> #pragma optimize("Ofast")
> ^
> /opt/deviceatlas/Src//dac.c:1065:0: warning: ignoring #pragma optimize
> [-Wunknown-pragmas]
> #pragma optimize("Ofast")
> ^
> /opt/deviceatlas/Src//dac.c:1076:0: warning: ignoring #pragma optimize
> [-Wunknown-pragmas]
> #pragma optimize("Ofast")
> ^
> /opt/deviceatlas/Src//dac.c: In function ârun_san_evidenceâ:
> /opt/deviceatlas/Src//dac.c:1404:23: warning: unused variable âeâ
> [-Wunused-variable]
> char *o, *e, *optr, *ptr;
>^
> /opt/deviceatlas/Src//dac.c:1393:22: warning: unused variable âeâ
> [-Wunused-variable]
> char *text, *e;
> ^
> /opt/deviceatlas/Src//dac.c: In function ârun_props_rulesâ:
> /opt/deviceatlas/Src//dac.c:2143:24: warning: missing braces around
> initializer [-Wmissing-braces]
> struct da_lkp_handler tmpch = {0};
> ^
> /opt/deviceatlas/Src//dac.c:2143:24: warning: (near initialization for
> âtmpch.evidâ) [-Wmissing-braces]
> /opt/deviceatlas/Src//dac.c:1984:19: warning: unused variable âeâ
> [-Wunused-variable]
> char *o, *e, *ptr, *optr;
>^
> /opt/deviceatlas/Src//dac.c:1983:38: warning: unused variable âedâ
> [-Wunused-variable]
> char *tmpkp, *tmpvp, *sptr, *ed;
> ^
> /opt/deviceatlas/Src//dac.c: In function ârun_dyn_rulesâ:
> /opt/deviceatlas/Src//dac.c:2192:13: error: âforâ loop initial
> declarations are only allowed in C99 mode
> for (size_t gt = 0; gt < result->fl[lvl]->yszcount; gt ++) {
> ^
> compilation terminated due to -Wfatal-errors.
> make: *** [/opt/deviceatlas/Src//dac.o] Error 1
>
>
> Additionally haproxy-2.8.3 version we check
> with deviceatlas-enterprise-c-2.4.1 version no issue observe , So let us
> know is there any concern with deviceatlas-enterprise-c-3.2 version
>
> --mahen
>
> On Mon, Sep 18, 2023 at 11:22 PM David CARLIER wrote:
>
>> Hi and thanks for your report.
>>
>> Would the following works for you (recommended to work a
>> fresh extracted directory) ?
>>
>> `cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2
>> -DCMAKE_C_FLAGS="-std=gnu99"`
>>
>>
>> On Mon, 18 Sept 2023 at 18:45, Amol Arote wrote:
>>
>>> Dear Team,
>>>
>>> We are trying to upgrade/install deviceatlas-enterprise-c-3.2 , but
>>> while compiling deviceatlas its showing some error.
>>> Below are the steps which we performed and occured error for the sam
Re: Error While deviceatlas compile/make from source
Hi and thanks for your report.
Would the following works for you (recommended to work a
fresh extracted directory) ?
`cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2
-DCMAKE_C_FLAGS="-std=gnu99"`
On Mon, 18 Sept 2023 at 18:45, Amol Arote wrote:
> Dear Team,
>
> We are trying to upgrade/install deviceatlas-enterprise-c-3.2 , but while
> compiling deviceatlas its showing some error.
> Below are the steps which we performed and occured error for the same.
>
> *Device Atlas Version :* deviceatlas-enterprise-c-3.2
>
> [root@govinda opt]# cd /opt/deviceatlas-enterprise-c-3.2/Src/
> [root@govinda Src]# cmake .
> -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2
> -- The C compiler identification is GNU 4.8.5
> -- The CXX compiler identification is GNU 4.8.5
> -- Check for working C compiler: /usr/bin/cc
> -- Check for working C compiler: /usr/bin/cc -- works
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Check for working CXX compiler: /usr/bin/c++
> -- Check for working CXX compiler: /usr/bin/c++ -- works
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Found CURL: /usr/lib64/libcurl.so (found version "7.29.0")
> -- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7")
> -- Performing Test HAVE_BUILTIN__BOOL
> -- Performing Test HAVE_BUILTIN__BOOL - Success
> -- Found OpenSSL: /usr/lib64/libssl.so;/usr/lib64/libcrypto.so (found
> version "1.0.2k")
> -- Found OpenSSL MD5
> -- Performing Test HAS_CURLSSLSET
> -- Performing Test HAS_CURLSSLSET - Failed
> -- Could NOT find ZIP
> -- Performing Test HAS_STD_ATOMICS
> -- Performing Test HAS_STD_ATOMICS - Failed
> -- Performing Test HAS_BUILTIN_ATOMICS
> -- Performing Test HAS_BUILTIN_ATOMICS - Success
> -- Performing Test HAS_ATTR_COLD
> -- Performing Test HAS_ATTR_COLD - Success
> -- Performing Test HAS_ATTR_ALLOC
> -- Performing Test HAS_ATTR_ALLOC - Failed
> -- Performing Test HAS_ATTR_NOSANITIZE
> -- Performing Test HAS_ATTR_NOSANITIZE - Success
> -- Performing Test HAS_BUILTIN_EXPECT
> -- Performing Test HAS_BUILTIN_EXPECT - Success
> -- Performing Test HAS_WIN32_ATOMICS
> -- Performing Test HAS_WIN32_ATOMICS - Failed
> -- Performing Test HAS_WIN32_ATTR_ALLOC
> -- Performing Test HAS_WIN32_ATTR_ALLOC - Failed
> -- Performing Test HAS_WIN32_UNUSED
> -- Performing Test HAS_WIN32_UNUSED - Failed
> -- version
> -- Configuring done
> -- Generating done
> -- Build files have been written to: /opt/deviceatlas-enterprise-c-3.2/Src
>
> [image: image.png]
>
>
> [root@govinda Src]# make
> Scanning dependencies of target ci
> [ 6%] Building C object CMakeFiles/ci.dir/ci.c.o
> [ 13%] Building C object CMakeFiles/ci.dir/dadwcom.c.o
> [ 20%] Building C object CMakeFiles/ci.dir/dadwcurl.c.o
> [ 26%] Building C object CMakeFiles/ci.dir/dadwarc.c.o
> [ 33%] Building C object CMakeFiles/ci.dir/cisch.c.o
> [ 40%] Building C object CMakeFiles/ci.dir/Os/ciunix.c.o
> Linking C shared library libci.so
> [ 40%] Built target ci
> Scanning dependencies of target da
> [ 46%] Building C object CMakeFiles/da.dir/Arch/amd64.c.o
> [ 53%] Building C object CMakeFiles/da.dir/json.c.o
> [ 60%] Building C object CMakeFiles/da.dir/dac.c.o
> /opt/deviceatlas-enterprise-c-3.2/Src/dac.c: In function ârun_dyn_rulesâ:
> /opt/deviceatlas-enterprise-c-3.2/Src/dac.c:2192:13: error: âforâ loop
> initial declarations are only allowed in C99 mode
> for (size_t gt = 0; gt < result->fl[lvl]->yszcount; gt ++) {
> ^
> /opt/deviceatlas-enterprise-c-3.2/Src/dac.c:2192:13: note: use option
> -std=c99 or -std=gnu99 to compile your code
> make[2]: *** [CMakeFiles/da.dir/dac.c.o] Error 1
> make[1]: *** [CMakeFiles/da.dir/all] Error 2
> make: *** [all] Error 2
> [root@govinda Src]#
>
> [image: image.png]
>
> Regards,
>
>
>
> Amol Arote
>
> Senior IT Manager
>
>
>
> *Mobile*: 9773868585 | 8097988585
>
> *Phone:* (022) 61934700 Ext 444
>
> *Email:* amol.ar...@naaptol.com
>
> *Web:* *https://www.naaptol.com *
>
>
>
>
Re: haproxy 2.4 and Kafka sink/source connector issues
We've tested 2.3.21 and 2.2.30 successfully, so it appears to be a 2.4 addition. We've tested 2.4.23 and the latest 2.7 and 2.8 versions. *David GreenwaldSenior Site Reliability engineerdavid.greenw...@discogsinc.com * On Tue, Aug 1, 2023 at 9:16 PM Willy Tarreau wrote: > On Tue, Aug 01, 2023 at 08:38:24PM -0700, David Greenwald wrote: > > Thanks for the response! That seems unlikely, we're doing an httpchk > > to the clustercheck > > utility > > < > https://docs.percona.com/percona-xtradb-cluster/5.7/howtos/virt_sandbox.html > > > > following the pxc reference architecture, so not actually making a direct > > database request from haproxy. We're also accessing the database with the > > same healthchecks from Python web applications without any issues, we're > > just seeing this from the long-lived connections, specifically JDBC sink > > connectors and Debezium as a source connector. > > > > This seems to be a pretty esoteric issue and we've come up empty in our > > Googling, unfortunately. > > OK. You said you encountered the problem when migrating to 2.4, what > was the last version you're aware of that didn't cause this problem ? > > Willy > -- The contents of this communication are confidential. If you are not the intended recipient, please immediately notify the sender by reply email and delete this message and its attachments, if any.
Re: haproxy 2.4 and Kafka sink/source connector issues
Thanks for the response! That seems unlikely, we're doing an httpchk to the clustercheck utility <https://docs.percona.com/percona-xtradb-cluster/5.7/howtos/virt_sandbox.html> following the pxc reference architecture, so not actually making a direct database request from haproxy. We're also accessing the database with the same healthchecks from Python web applications without any issues, we're just seeing this from the long-lived connections, specifically JDBC sink connectors and Debezium as a source connector. This seems to be a pretty esoteric issue and we've come up empty in our Googling, unfortunately. *David GreenwaldSenior Site Reliability engineerdavid.greenw...@discogsinc.com * On Tue, Aug 1, 2023 at 8:25 PM Willy Tarreau wrote: > Hi David, > > On Tue, Aug 01, 2023 at 05:11:48PM -0700, David Greenwald wrote: > > Hi all, > > > > Looking for some help with a networking issue we've been debugging for > > several days. We use haproxy to TCP load-balance between Kafka Connectors > > and a Percona MySQL cluster. In this set-up, the connectors (i.e., Java > > JDBC) maintain long-running connections and read the database binlogs. > This > > includes regular polling. > > > > We have seen connection issues starting with haproxy 2.4 and persisting > > through 2.8 which result in the following errors in MySQL: > > > > 2023-07-31T17:25:45.745607Z 3364649 [Note] Got an error reading > > communication packets > > > > As you can see, this doesn't include a host or user and is happening > early > > in the connection. The host cache shows handshake errors here regularly > > accumulating. > > > > We were unable to see errors on the haproxy side with tcplog on and have > > been unable to get useful information from tcpdump, netstat, etc. > > > > We are aware FE/BE connection closure behavior changed in 2.4. The 2.4 > > option of idle-close-on-response seemed like a possible solution but > isn't > > compatible with mode tcp, so we're not sure what's happening here or next > > steps for debugging. Appreciate any help or guidance here. > > The changes you're referring to are indeed only in HTTP mode. Have you > tried without health checks ? I'm indeed wondering if the handshake errors > you're observing are not simply health checks, which would explain why > you're not seeing them in your traffic logs. > > Willy > -- The contents of this communication are confidential. If you are not the intended recipient, please immediately notify the sender by reply email and delete this message and its attachments, if any.
haproxy 2.4 and Kafka sink/source connector issues
Hi all, Looking for some help with a networking issue we've been debugging for several days. We use haproxy to TCP load-balance between Kafka Connectors and a Percona MySQL cluster. In this set-up, the connectors (i.e., Java JDBC) maintain long-running connections and read the database binlogs. This includes regular polling. We have seen connection issues starting with haproxy 2.4 and persisting through 2.8 which result in the following errors in MySQL: 2023-07-31T17:25:45.745607Z 3364649 [Note] Got an error reading communication packets As you can see, this doesn't include a host or user and is happening early in the connection. The host cache shows handshake errors here regularly accumulating. We were unable to see errors on the haproxy side with tcplog on and have been unable to get useful information from tcpdump, netstat, etc. We are aware FE/BE connection closure behavior changed in 2.4. The 2.4 option of idle-close-on-response seemed like a possible solution but isn't compatible with mode tcp, so we're not sure what's happening here or next steps for debugging. Appreciate any help or guidance here. We're running haproxy in Kubernetes using the official container, and are also not seeing any issues with current haproxy versions with our other (Python) applications. A simplified version of our config: global daemon maxconn 25000 defaults balance roundrobin option dontlognull option redispatch timeout http-request 5s timeout queue 1m timeout connect 4s timeout client 50s timeout server 30s timeout http-keep-alive 10s timeout check 10s retries 3 frontend main_writer bind :3306 mode tcp timeout client 30s timeout client-fin 30s default_backend main_writer backend main_writer mode tcp balance leastconn option httpchk timeout server 30s timeout tunnel 3h server db1 :3306 check port 9200 on-marked-down shutdown-sessions weight 100 inter 3s rise 1 fall 2 server db2 :3306 check port 9200 on-marked-down shutdown-sessions weight 100 backup server db3 :3306 check port 9200 on-marked-down shutdown-sessions weight 100 backup *David GreenwaldSenior Site Reliability engineerdavid.greenw...@discogsinc.com * -- The contents of this communication are confidential. If you are not the intended recipient, please immediately notify the sender by reply email and delete this message and its attachments, if any.
Re: [PATCH] BUILD/MINOR da
Here another revised version of the patch. Kindest regards. On Tue, Mar 28, 2023 at 6:41 AM David Carlier wrote: > Sorry here a corrected version. > > Cheers. > > On Tue, Mar 28, 2023 at 6:30 AM David Carlier > wrote: > >> Hi, >> >> Here a minor build update for the da vendor module. >> >> To be backported to previous versions too. >> >> Thanks ! >> > 0001-BUILD-da-extends-CFLAGS-to-support-API-v3-from-3.1.7.patch Description: Binary data
Re: [PATCH] BUILD/MINOR da
Sorry here a corrected version. Cheers. On Tue, Mar 28, 2023 at 6:30 AM David Carlier wrote: > Hi, > > Here a minor build update for the da vendor module. > > To be backported to previous versions too. > > Thanks ! > 0001-BUILD-MINOR-da.patch Description: Binary data
[PATCH] BUILD/MINOR da
Hi, Here a minor build update for the da vendor module. To be backported to previous versions too. Thanks ! 0001-BUILD-MINOR-da.patch Description: Binary data
[PATCH]: thread disable thread affinity for macOs arm64
Hi, here a patch to disable the cpu affinity feature on macOs arm64, see the comments for more explanation. Cheers ! 0001-BUILD-thread-disable-cpu-affinity-on-macOs-arm64.patch Description: Binary data
Re: [PATCH]: BUILD: insecure-setuid-wanted support on FreeBSD
ping :) is the mailing list still the way for patches or is github more appropriate for better traceability ? On Mon, 7 Nov 2022 at 13:37, David CARLIER wrote: > I sent a new one after, resending in case it got lost. > > On Mon, 7 Nov 2022 at 12:36, Tim Düsterhus wrote: > >> David, >> >> On 11/7/22 09:00, David CARLIER wrote: >> > Thanks here a corrected version. >> > >> >> It looks like you accidentally attached the same patch as before. >> >> Best regards >> Tim Düsterhus >> >
Re: [PATCH]: BUILD: insecure-setuid-wanted support on FreeBSD
I sent a new one after, resending in case it got lost. On Mon, 7 Nov 2022 at 12:36, Tim Düsterhus wrote: > David, > > On 11/7/22 09:00, David CARLIER wrote: > > Thanks here a corrected version. > > > > It looks like you accidentally attached the same patch as before. > > Best regards > Tim Düsterhus > 0001-BUILD-insecure-setuid-wanted-support-on-FreeBSD.patch Description: Binary data
Re: [PATCH]: BUILD: insecure-setuid-wanted support on FreeBSD
Thanks here a corrected version.
Regards.
On Mon, 7 Nov 2022 at 07:01, Willy Tarreau wrote:
> Hi David,
>
> On Fri, Nov 04, 2022 at 07:34:46PM +0000, David CARLIER wrote:
> > Hi,
> >
> > here a little patch to port the insecure-setuid-wanted directive on
> FreeBSD.
>
> Thanks. I'm having a few comments below.
>
> > From be693024d7e49173f7ff37566232238fc5ea1887 Mon Sep 17 00:00:00 2001
> > From: David CARLIER
> > Date: Fri, 4 Nov 2022 19:24:03 +
> > Subject: [PATCH] BUILD: insecure-setuid-wanted support on FreeBSD.
> >
> > using the procctl api to ignore the suid/sgid bits to be ignored.
> > ---
> > src/haproxy.c | 11 +--
> > 1 file changed, 9 insertions(+), 2 deletions(-)
> >
> > diff --git a/src/haproxy.c b/src/haproxy.c
> > index 806497062..94b9bde4e 100644
> > --- a/src/haproxy.c
> > +++ b/src/haproxy.c
> > @@ -3003,7 +3003,7 @@ static void *run_thread_poll_loop(void *data)
> > pthread_mutex_unlock(_mutex);
> > #endif
> >
> > -#if defined(PR_SET_NO_NEW_PRIVS) && defined(USE_PRCTL)
> > +#if (defined(PR_SET_NO_NEW_PRIVS) && defined(USE_PRCTL)) ||
> (defined(PROC_NO_NEW_PRIVS_CTL) && defined(USE_PROCCTL))
>
> At this point this should move to compat.h, to do something like this
> for example:
>
> #if (defined(PR_SET_NO_NEW_PRIVS) && defined(USE_PRCTL)) || \ /*
> linux */
> (defined(PROC_NO_NEW_PRIVS_CTL) && defined(USE_PROCCTL)) /*
> freebsd */
> #define HAVE_NO_NEW_PRIVS
> #endif
>
> Then here you'd just use:
>
> #ifdef HAVE_NO_NEW_PRIVS
>
> > /* Let's refrain from using setuid executables. This way the
> impact of
> >* an eventual vulnerability in a library remains limited. It may
> >* impact external checks but who cares about them anyway ? In the
> > @@ -3014,7 +3014,14 @@ static void *run_thread_poll_loop(void *data)
> >*/
> > if (!(global.tune.options & GTUNE_INSECURE_SETUID) && !master) {
> > static int warn_fail;
> > - if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) == -1 &&
> !_HA_ATOMIC_FETCH_ADD(_fail, 1)) {
> > +#if defined(USE_PRCTL)
> > + if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) == -1
> > +#else
> > + int refrain_setuid = PROC_NO_NEW_PRIVS_ENABLE;
> > + /* we can save one syscall once freebsd 14 becomes the
> minimum version, removing getpid */
> > + if (procctl(P_PID, getpid(), PROC_NO_NEW_PRIVS_CTL,
> _setuid) == -1
> > +#endif
> > + && !_HA_ATOMIC_FETCH_ADD(_fail, 1)) {
>
> And the part above is not nice, it breaks an "if" expression making it
> much harder to follow, which is particularly problematic in code aimed
> at improving security. I wouldn't be surprised if it would even break
> auto-indent or parens matching on some editors.
>
> Better put that in a separate function such as ha_no_new_privs() which
> calls the suitable syscall and arguments depending on which approach is
> desired. It could be left in this file I guess, even though for the long
> term I think we'll finally create an "src/sys.c" file to support system
> specific abstraction but we'd rather do that later.
>
> Thanks,
> Willy
>
0001-BUILD-insecure-setuid-wanted-support-on-FreeBSD.patch
Description: Binary data
[PATCH]: BUILD: insecure-setuid-wanted support on FreeBSD
Hi, here a little patch to port the insecure-setuid-wanted directive on FreeBSD. Cheers. 0001-BUILD-insecure-setuid-wanted-support-on-FreeBSD.patch Description: Binary data
Re: [PATCH 1/1]: MINOR __builtin_memcpy_inline usage introduction
On Mon, 20 Jun 2022 at 07:27, Willy Tarreau wrote:
>
> Hi David,
>
> On Sat, Jun 18, 2022 at 12:52:23PM +0100, David CARLIER wrote:
> > From 9d7b6448a2407451c3115b701c51f97ab2bf6a59 Mon Sep 17 00:00:00 2001
> > From: David Carlier
> > Date: Sat, 18 Jun 2022 12:41:11 +0100
> > Subject: [PATCH] MINOR: compiler __builtin_memcpy_inline usage introduction.
> >
> > Optimised version of the existing __builtin_memcpy builtin, differences
> > reside by the fact it works only with constant time sizes and does
> > generate extra calls.
> > At the moment, is clang exclusive even tough GCC does not seem to
> > want to implement it, the comments try not to reflect this current
> > state.
> > Usage can be expanded, used purposely only in few places for starter.
> > ---
> > include/haproxy/compiler.h | 12
> > src/haproxy.c | 4 ++--
> > src/log.c | 2 +-
> > src/proxy.c| 2 +-
> > 4 files changed, 16 insertions(+), 4 deletions(-)
> >
> > diff --git a/include/haproxy/compiler.h b/include/haproxy/compiler.h
> > index 66b5f5835..dca9f6aef 100644
> > --- a/include/haproxy/compiler.h
> > +++ b/include/haproxy/compiler.h
> > @@ -192,6 +192,18 @@
> > #endif
> > #endif
> >
> > +/*
> > + * This builtin works only with compile time lengths unlike
> > __builtin_memcpy.
> > + * Also guarantees no external call is generated.
> > + * We could `replicate` the aforementioned constraint with a
> > + * _Static_assert/__builtin_constant_p theoretically, but that might be
> > + * too much trouble to make it happens (C11 min)
> > + * so here we trust the proper usage with other compilers (and the CI
> > infrastructure).
> > + */
> > +#if !defined(__clang__) || __clang_major__ < 11
> > +#define __builtin_memcpy_inline(x, y, s) memcpy(x, y, s)
> > +#endif
> > +
>
> Hmmm please no, this is not the right way to do it for two reasons:
> - it will encourage use of __builtin_memcpy_inline() making one believe
> it's the expected one when it isn't ;
>
> - developing on non-clang systems will not report the problem with the
> non-constant size that __builtin_memcpy_inline() expects, resulting
> in breakage from time to time.
>
> I'd suggest that instead you create a macro named ha_memcpy_inline() that
> maps to __builtin_memcpy_inline() when present and s is a builtin constant,
> or maps to __builtin_memcpy() for the rest. Please note, by the way, that
> gcc since at least 3.4 has been emitting the same instructions (rep movsb)
> as clang's __builtin_memcpy_inline(), but only when optimizing for size.
> When optimizing for anything else, it can emit ton of inlined garbage^Wvector
> instructions.
>
> Thus I suspect we could achieve the same result by doing a clever
> arrangement of #pragma GCC push_options / #pragma GCC optimize("Os,inline")
> around an inline function definition that does the memcpy, and that is
> called by the macro. I have not tried, but probably something like this
> would do it:
>
> #pragma GCC push_options
> #pragma GCC optimize("Os,inline")
> static inline void _ha_memcpy_inline(void *restrict dest, const void
> *restrict src, size_t n)
> {
> __builtin_memcpy(dest, src, n);
> }
> #pragma GCC pop_options
>
> #if defined(clang...)
> #define ha_memcpy_inline(d,s,n) do { if (__builtin_constant_p(n))
> __builtin_memcpy_inline(d,s,n); else _ha_memcpy_inline(d,s,n); } while (0)
> #else
> #define ha_memcpy_inline(d,s,n) _ha_memcpy_inline(d,s,n)
> #endif
>
> That's not even compile-tested and I haven't looked at the result, but
> you get the idea.
Alright thanks here a newer version.
>
> Now regarding where to use it... I'd say it depends on a lot of factors,
> size, speed, undesired dependency on external libs. I think that each and
> every call place does warrant an individual commit with a justification
> and a check that the benefit matches expectations. There could be some
> value in using this in various low-level protocol layers (QUIC, HTX, SPOE).
Exactly.
> Maybe more, I don't know.
Over time its value will display more.
>
> Thanks,
> Willy
From 90c951d29c32a345cf38d6c0b43ee904f12b8ac7 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Sat, 18 Jun 2022 12:41:11 +0100
Subject: [PATCH] MINOR: compiler __builtin_memcpy_inline usage introduction.
Optimised version of the existing __builtin_memcpy builtin, differences
reside by the fact it works only with constant time sizes and does
not generate extra calls.
At the moment, is clang exclusive even tough GCC does not seem to
want to implement it, the
[PATCH 1/1]: MINOR __builtin_memcpy_inline usage introduction
Hi here a little proposal to use this specific clang builtin, only placed in few places but over time can be expanded cautiously. Kindest regards. Thanks. From 9d7b6448a2407451c3115b701c51f97ab2bf6a59 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Sat, 18 Jun 2022 12:41:11 +0100 Subject: [PATCH] MINOR: compiler __builtin_memcpy_inline usage introduction. Optimised version of the existing __builtin_memcpy builtin, differences reside by the fact it works only with constant time sizes and does generate extra calls. At the moment, is clang exclusive even tough GCC does not seem to want to implement it, the comments try not to reflect this current state. Usage can be expanded, used purposely only in few places for starter. --- include/haproxy/compiler.h | 12 src/haproxy.c | 4 ++-- src/log.c | 2 +- src/proxy.c| 2 +- 4 files changed, 16 insertions(+), 4 deletions(-) diff --git a/include/haproxy/compiler.h b/include/haproxy/compiler.h index 66b5f5835..dca9f6aef 100644 --- a/include/haproxy/compiler.h +++ b/include/haproxy/compiler.h @@ -192,6 +192,18 @@ #endif #endif +/* + * This builtin works only with compile time lengths unlike __builtin_memcpy. + * Also guarantees no external call is generated. + * We could `replicate` the aforementioned constraint with a + * _Static_assert/__builtin_constant_p theoretically, but that might be + * too much trouble to make it happens (C11 min) + * so here we trust the proper usage with other compilers (and the CI infrastructure). + */ +#if !defined(__clang__) || __clang_major__ < 11 +#define __builtin_memcpy_inline(x, y, s) memcpy(x, y, s) +#endif + /* Linux-like "container_of". It returns a pointer to the structure of type * which has its member stored at address . */ diff --git a/src/haproxy.c b/src/haproxy.c index 32eb4bdc6..1464d29ee 100644 --- a/src/haproxy.c +++ b/src/haproxy.c @@ -1265,11 +1265,11 @@ static void ha_random_boot(char *const *argv) /* stack address (benefit form operating system's ASLR) */ l = (unsigned long) - memcpy(m, , sizeof(l)); m += sizeof(l); + __builtin_memcpy_inline(m, , sizeof(l)); m += sizeof(l); /* argv address (benefit form operating system's ASLR) */ l = (unsigned long) - memcpy(m, , sizeof(l)); m += sizeof(l); + __builtin_memcpy_inline(m, , sizeof(l)); m += sizeof(l); /* use tv_usec again after all the operations above */ gettimeofday(, NULL); diff --git a/src/log.c b/src/log.c index 304e3cb68..a631b1e46 100644 --- a/src/log.c +++ b/src/log.c @@ -808,7 +808,7 @@ int parse_logsrv(char **args, struct list *logsrvs, int do_del, const char *file } node = malloc(sizeof(*node)); - memcpy(node, logsrv, sizeof(struct logsrv)); + __builtin_memcpy_inline(node, logsrv, sizeof(struct logsrv)); node->ref = logsrv; LIST_INIT(>list); LIST_APPEND(logsrvs, >list); diff --git a/src/proxy.c b/src/proxy.c index c10a8cf83..7d02f8c19 100644 --- a/src/proxy.c +++ b/src/proxy.c @@ -1818,7 +1818,7 @@ static int proxy_defproxy_cpy(struct proxy *curproxy, const struct proxy *defpro memprintf(errmsg, "proxy '%s': out of memory", curproxy->id); return 1; } - memcpy(node, tmplogsrv, sizeof(struct logsrv)); + __builtin_memcpy_inline(node, tmplogsrv, sizeof(struct logsrv)); node->ref = tmplogsrv->ref; LIST_INIT(>list); LIST_APPEND(>logsrvs, >list); -- 2.34.1
Grow Brand Awareness, Engagement & Traffic.
Dear Business owner of haproxy.org*,* Grow Brand Awareness, Engagement & Traffic. Our social media marketing campaigns start with a purpose. A goal. To drive customers, grow your audience and expand your reach. Our social media management covers all aspects of developing your social presence while showcasing your company’s personality. *get started at just $49 for any social media account for a trial.* Kind Regards, *David* *Note:* - Our next conversation will be on my corporate Email ID. Reply me with "*Quote*", if you are interested. [image: beacon]
Re: Segfault on 2.6.0 with TCP switching to HTTP/2
On Thu, 16 Jun 2022 at 20:27, Aleksandar Lazic wrote: [...] > > Thanks ! I'm able to reproduce the segfault. I'm on it. Thanks! > But in any way wouldn't be better that the rule > > acl ipwtf hdr(Host),lower,field(1,:),word(-1,.,2) ip.wtf > > be after > > > >tcp-request inspect-delay 10s > > >tcp-request content switch-mode http if !ipwtf > > because it "feels somehow wrong" to make header checks in tcp mode. There's some explanation in the configuration manual about how it works, and it's documented to work, at least for HTTP/1. https://docs.haproxy.org/2.6/configuration.html#4 "While HAProxy is able to parse HTTP/1 in-fly from tcp-request content rules"... Essentially I want to keep the connection as TCP, so that I can have a backend that gets raw HTTP/1.1. I wrote some more about it at https://dgl.cx/2022/04/showing-you-your-actual-http-request [...] > Opinions? Clearly in nearly all cases it's better to let haproxy be the HTTP proxy layer, especially as it isn't possible to mix for HTTP/2, but it lets me do my crazy thing here :) David
Segfault on 2.6.0 with TCP switching to HTTP/2
I tried upgrading to 2.6.0 (from 2.5.6) and I'm seeing a segfault when making HTTP/2 requests. I'm using a frontend in TCP mode and then switching it to HTTP/2. I've made a minimal config that exhibits the segfault, below. Simply doing curl -vk https://ip is enough to trigger it for me. Thread 1 "haproxy" received signal SIGSEGV, Segmentation fault. 0x555d1d07 in h2s_close (h2s=0x55a60b70) at src/mux_h2.c:1497 1497 HA_ATOMIC_DEC(>h2c->px_counters->open_streams); (gdb) bt #0 0x555d1d07 in h2s_close (h2s=0x55a60b70) at src/mux_h2.c:1497 #1 h2s_destroy (h2s=0x55a60b70) at src/mux_h2.c:1515 #2 0x555d3463 in h2_detach (sd=) at src/mux_h2.c:4432 The exact backtrace varies but always in h2s_destroy. (In case you're wondering what on earth I'm doing, there's a write-up of it at https://dgl.cx/2022/04/showing-you-your-actual-http-request) David --- global ssl-default-bind-options no-sslv3 no-tlsv10 user nobody defaults timeout connect 10s timeout client 30s timeout server 2m frontend tcp-https mode tcp bind [::]:443 v4v6 ssl crt /etc/haproxy/ssl/bodge.cloud.pem alpn h2,http/1.1 acl ipwtf hdr(Host),lower,field(1,:),word(-1,.,2) ip.wtf default_backend ipwtf tcp-request inspect-delay 10s tcp-request content switch-mode http if !ipwtf use_backend cloud-regions.bodge.cloud if !ipwtf backend ipwtf mode tcp server ipwtf localhost:8080 backend cloud-regions.bodge.cloud mode http server cr localhost:8080
Re: deviceatlas compiler error
My pleasure Amol, So for the remaining "failures", if this is those you mention Performing Test HAS_STD_ATOMICS -- Performing Test HAS_STD_ATOMICS - Failed -- Performing Test HAS_BUILTINS_ATOMICS -- Performing Test HAS_BUILTINS_ATOMICS - Success -- Performing Test HAS_ATTR_COLD -- Performing Test HAS_ATTR_COLD - Success -- Performing Test HAS_ATTR_ALLOC -- Performing Test HAS_ATTR_ALLOC - Failed -- Performing Test HAS_WIN32_ATOMICS -- Performing Test HAS_WIN32_ATOMICS - Failed -- Performing Test HAS_WIN32_ATTR_ALLOC -- Performing Test HAS_WIN32_ATTR_ALLOC - Failed -- Performing Test HAS_WIN32_UNUSED -- Performing Test HAS_WIN32_UNUSED - Failed They are normal and expected since you re using gcc 4.8 and being on an unix system. I would suggest however typing this cmake command instead cmake -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas -DCMAKE_BUILD_TYPE=Release then when you ll type make, normally the api should compile. Kindest regards. On Fri, 3 Jun 2022 at 16:58, Amol Arote wrote: > Thank you sir , For Prompt Reply > > I install Some dependancies on CentOS 7.6 as per your last update > libcurl-devel , libzip-devel > But Seen Some failed Test Below , So let us know what Further Changes to > be done > OR can Ignore these Failed Test > > > > > --- > Error while compiling > > --- > > > [root@test Src]# cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas > -- The C compiler identification is GNU 4.8.5 > -- The CXX compiler identification is GNU 4.8.5 > -- Check for working C compiler: /usr/bin/cc > -- Check for working C compiler: /usr/bin/cc -- works > -- Detecting C compiler ABI info > -- Detecting C compiler ABI info - done > -- Check for working CXX compiler: /usr/bin/c++ > -- Check for working CXX compiler: /usr/bin/c++ -- works > -- Detecting CXX compiler ABI info > -- Detecting CXX compiler ABI info - done > -- Found PCRE: /usr/include > -- Found CURL: /usr/lib64/libcurl.so (found version "7.29.0") > -- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7") > -- Performing Test HAVE_BUILTIN__BOOL > -- Performing Test HAVE_BUILTIN__BOOL - Success > -- Found OpenSSL: /usr/lib64/libssl.so;/usr/lib64/libcrypto.so (found > version "1.0.2k") > -- Found OpenSSL MD5 > -- Performing Test HAS_CURLSSLSET > -- Performing Test HAS_CURLSSLSET - Failed > -- Found ZIP: /usr/lib64/libzip.so > -- Performing Test HAS_STD_ATOMICS > -- Performing Test HAS_STD_ATOMICS - Failed > -- Performing Test HAS_BUILTINS_ATOMICS > -- Performing Test HAS_BUILTINS_ATOMICS - Success > -- Performing Test HAS_ATTR_COLD > -- Performing Test HAS_ATTR_COLD - Success > -- Performing Test HAS_ATTR_ALLOC > -- Performing Test HAS_ATTR_ALLOC - Failed > -- Performing Test HAS_WIN32_ATOMICS > -- Performing Test HAS_WIN32_ATOMICS - Failed > -- Performing Test HAS_WIN32_ATTR_ALLOC > -- Performing Test HAS_WIN32_ATTR_ALLOC - Failed > -- Performing Test HAS_WIN32_UNUSED > -- Performing Test HAS_WIN32_UNUSED - Failed > -- version > -- Configuring done > -- Generating done > -- Build files have been written to: /opt/deviceatlas/Src > > > > Regards, > > > > Amol Arote > > Senior IT Manager > > > > *Mobile*: 9773868585 | 8097988585 > > *Phone:* (022) 61934700 Ext 444 > > *Email:* amol.ar...@naaptol.com > > *Web:* *https://www.naaptol.com <https://www.naaptol.com>* > > > > > On Fri, Jun 3, 2022 at 12:25 PM David CARLIER wrote: > >> Hi Amole and thanks for your report. >> >> The C api 2.4.0 version is a major upgrade which comes with additional >> dependencies. >> Indeed as mentioned by your report >> >> ... >> > > -- Could NOT find CURL (missing: CURL_LIBRARY CURL_INCLUDE_DIR) >> ... >> > > -- Could NOT find ZIP >> ... >> >> Here a sample of the README.Unix.html doc page >> >> RedHat/CentOS >> CentOS < 8 version >> Note: the libzip-devel system package is obsolete, thus the >> libzip-last-devel >> ought to be used instead. A third party repository might need to be >> enabled. >> ```shell >> % sudo dnf install gcc (or clang) pcre-devel cmake zlib-devel >> libzip-last-devel \ >> libcurl-devel >> ``` >> CentOS >= 8 version >> ```shell >> % sudo dnf install gcc (or clang) pcre-devel cmake zlib-devel >> libzip-devel \ >> libcurl-devel >> ``` >> >> If you have any further question, please let me know. >> >> Kindest regards. >> >> On Fri, 3 Jun 2022 at 07:21
Re: deviceatlas compiler error
Hi Amole and thanks for your report. The C api 2.4.0 version is a major upgrade which comes with additional dependencies. Indeed as mentioned by your report ... > > -- Could NOT find CURL (missing: CURL_LIBRARY CURL_INCLUDE_DIR) ... > > -- Could NOT find ZIP ... Here a sample of the README.Unix.html doc page RedHat/CentOS CentOS < 8 version Note: the libzip-devel system package is obsolete, thus the libzip-last-devel ought to be used instead. A third party repository might need to be enabled. ```shell % sudo dnf install gcc (or clang) pcre-devel cmake zlib-devel libzip-last-devel \ libcurl-devel ``` CentOS >= 8 version ```shell % sudo dnf install gcc (or clang) pcre-devel cmake zlib-devel libzip-devel \ libcurl-devel ``` If you have any further question, please let me know. Kindest regards. On Fri, 3 Jun 2022 at 07:21, Willy Tarreau wrote: > > Hello Amol, > > On Fri, Jun 03, 2022 at 11:09:07AM +0530, Amol Arote wrote: > > David, please find the rest of the report below. > > Thanks! > Willy > > > --- > > *Versions* > > --- > > HAProxy version 2.4.2-553dee3 2021/07/07 > > cmake version 2.8.12.2 > > CentOS Linux release 7.6.1810 (Core) > > deviceatlas-enterprise-c-2.4.0.zip > > --- > > *deviceatlas compile steps* > > --- > > # yum install cmake > > # unzip deviceatlas-enterprise-c-2.4.0.zip > > # mv deviceatlas-enterprise-c-2.4.0 deviceatlas > > # cd /opt/deviceatlas/Src/ > > # cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas > > # make > > --- > > *Error while compiling woth command * > > * # cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas * > > --- > > [root@tt Src]## cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas > > -- The C compiler identification is GNU 4.8.5 > > -- The CXX compiler identification is GNU 4.8.5 > > -- Check for working C compiler: /usr/bin/cc > > -- Check for working C compiler: /usr/bin/cc -- works > > -- Detecting C compiler ABI info > > -- Detecting C compiler ABI info - done > > -- Check for working CXX compiler: /usr/bin/c++ > > -- Check for working CXX compiler: /usr/bin/c++ -- works > > -- Detecting CXX compiler ABI info > > -- Detecting CXX compiler ABI info - done > > -- Found PCRE: /usr/include > > -- Could NOT find CURL (missing: CURL_LIBRARY CURL_INCLUDE_DIR) > > -- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7") > > -- Performing Test HAVE_BUILTIN__BOOL > > -- Performing Test HAVE_BUILTIN__BOOL - Success > > -- Could NOT find ZIP > > -- Performing Test HAS_STD_ATOMICS > > -- Performing Test HAS_STD_ATOMICS - Failed > > -- Performing Test HAS_BUILTINS_ATOMICS > > -- Performing Test HAS_BUILTINS_ATOMICS - Success > > -- Performing Test HAS_ATTR_COLD > > -- Performing Test HAS_ATTR_COLD - Success > > -- Performing Test HAS_ATTR_ALLOC > > -- Performing Test HAS_ATTR_ALLOC - Failed > > -- Performing Test HAS_WIN32_ATOMICS > > -- Performing Test HAS_WIN32_ATOMICS - Failed > > -- Performing Test HAS_WIN32_ATTR_ALLOC > > -- Performing Test HAS_WIN32_ATTR_ALLOC - Failed > > -- Performing Test HAS_WIN32_UNUSED > > -- Performing Test HAS_WIN32_UNUSED - Failed > > -- version > > -- Configuring done > > -- Generating done > > -- Build files have been written to: /opt/deviceatlas/Src > > [root@tt Src]# # > > > > Request you to please guide us on above matter > > > > > > -- > > > > Regards, > > > > > > > > Amol Arote > > > > Senior IT Manager > > > > > > > > *Mobile*: 9773868585 | 8097988585 > > > > *Phone:* (022) 61934700 Ext 444 > > > > *Email:* amol.ar...@naaptol.com > > > > *Web:* *https://www.naaptol.com <https://www.naaptol.com>* > > > > -- > >
[PATCH 1/1] : BUILD/MINOR cpuset build fix for FreeBSD 13.1
Hi, FreeBSD 13.1 had been released this week and here a little fix for the cpuset part. Kind regards. 0001-BUILD-MINOR-cpuset-fix-build-for-FreeBSD-13.1.patch Description: Binary data
Re: [PATCH 1/1]: BUILD/MINOR: solaris based oses build fix/get_exe_path implementation.
sure. Regards. On Sat, 14 May 2022 at 16:32, Willy Tarreau wrote: > > Hi David, > > > From 5b175adfa5ef9ab52ce69f7eb6775efe8a828974 Mon Sep 17 00:00:00 2001 > > From: David Carlier > > Date: Fri, 13 May 2022 20:16:15 +0100 > > Subject: [PATCH] BUILD/MINOR: few solaris updates. > > > > - get_exec_path using getexecname, fetching AT_SUN_EXECNAME from the > > auxiliary vectors. > > - __maybe_unused already defined. > > Please split it in two, as it really addresses two completely independent > things. It seems that the former improves error reporting while the > second one is a build fix. > > Thanks, > Willy From fdc0ec787c050a05bac7d4ae5fdd9d159c38ad16 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Sat, 14 May 2022 17:10:50 +0100 Subject: [PATCH] BUILD/MINOR: fix build warning on solaris based systems. __maybe_unused is already defined. --- include/haproxy/compiler.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/haproxy/compiler.h b/include/haproxy/compiler.h index 49356daab..a935ac3b5 100644 --- a/include/haproxy/compiler.h +++ b/include/haproxy/compiler.h @@ -58,10 +58,12 @@ #endif #endif +#ifndef __maybe_unused /* silence the "unused" warnings without having to place painful #ifdefs. * For use with variables or functions. */ #define __maybe_unused __attribute__((unused)) +#endif /* These macros are used to declare a section name for a variable. * WARNING: keep section names short, as MacOS limits them to 16 characters. -- 2.34.1 From 765054a26a58866a7a9d290597e375f9f077b3f2 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Sat, 14 May 2022 17:15:49 +0100 Subject: [PATCH] BUILD/MINOR: get_exec_path implementation for solaris based systems. using getexecname which fetches AT_SUN_EXECNAME from the auxiliary vectors. --- src/tools.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/tools.c b/src/tools.c index ed3c3a667..c6bc81be9 100644 --- a/src/tools.c +++ b/src/tools.c @@ -4821,6 +4821,8 @@ const char *get_exec_path() break; } } +#elif defined(__sun) + ret = getexecname(); #endif return ret; } -- 2.34.1
[PATCH 1/1]: BUILD/MINOR: solaris based oses build fix/get_exe_path implementation.
Hi, here a little path for illumos/solaris. cheers. regards. From 5b175adfa5ef9ab52ce69f7eb6775efe8a828974 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Fri, 13 May 2022 20:16:15 +0100 Subject: [PATCH] BUILD/MINOR: few solaris updates. - get_exec_path using getexecname, fetching AT_SUN_EXECNAME from the auxiliary vectors. - __maybe_unused already defined. --- include/haproxy/compiler.h | 2 ++ src/tools.c| 2 ++ 2 files changed, 4 insertions(+) diff --git a/include/haproxy/compiler.h b/include/haproxy/compiler.h index 49356daab..a935ac3b5 100644 --- a/include/haproxy/compiler.h +++ b/include/haproxy/compiler.h @@ -58,10 +58,12 @@ #endif #endif +#ifndef __maybe_unused /* silence the "unused" warnings without having to place painful #ifdefs. * For use with variables or functions. */ #define __maybe_unused __attribute__((unused)) +#endif /* These macros are used to declare a section name for a variable. * WARNING: keep section names short, as MacOS limits them to 16 characters. diff --git a/src/tools.c b/src/tools.c index ed3c3a667..c6bc81be9 100644 --- a/src/tools.c +++ b/src/tools.c @@ -4821,6 +4821,8 @@ const char *get_exec_path() break; } } +#elif defined(__sun) + ret = getexecname(); #endif return ret; } -- 2.36.0
Re: [PATCH 1/1: BUILD/MINOR: TCP_KEEPIDLE macos equivalence
On Sun, 8 May 2022 at 09:57, Willy Tarreau wrote:
>
> On Sun, May 01, 2022 at 03:33:17PM +0100, David CARLIER wrote:
> > Hi here a little patch to set idle time for SO_KEEPALIVE socket option.
>
> Now merged, thanks.
>
> David, one comment though, your commit messages keep missing a lot of
> crucial information for reviewers and debuggers, and I had to spend time
> documenting myself on keep-alive on MacOS to figure the differences and/or
> what the impacts or limitations of the patch could be. I finally found and
> put that information into the commit message, but it would be much easier
> if you could put it yourself after your development since you actually
> have access to the information I had to seek, and know the reasoning
> behind your choice.
>
Thanks, fair point :-) I ll take it in account even tough I was
certain for this one it would not break anything.
...
> The new choice was explained again. Finally, two weeks later I got a
> report of breakage when using external code linked at run time, the
> keyword registration didn't work anymore due to a mistake in this last
> patch. Fortunately, the design decision was explained and I could
> restart from this, figure my mistake and make sure that the 3rd attempt
> at a fix this time addressed all 3 identified use cases:
Do you have a reasonable numbers of macOs users or is it a rare occurrence ?
>
> commit 65d9f83794e00e136335348de531167f31d2f39b
> Author: Willy Tarreau
> Date: Tue Apr 26 19:35:38 2022 +0200
>
> BUILD: compiler: properly distinguish weak and global symbols
>
> While weak symbols were finally fixed with commit fb1b6f5bc ("BUILD:
> compiler: use a more portable set of asm(".weak") statements"), it
> was an error to think that initcall symbols were also weak. They must
> not be and they're only global. The reason is that any externally
> linked code loaded as a .so would drop its weak symbols when being
> loaded, hence its initcalls that may contain various function
> registration calls.
>
> The ambiguity came from the fact that we initially reused the initcall's
> HA_GLOBL macro for OSX then generalized it, then turned it to a choice
> between .globl and .weak based on the OS, while in fact we needed a
> macro to define weak symbols.
>
> Let's rename the macro to HA_WEAK() to make it clear it's only for weak
> symbols, and redefine HA_GLOBL() that initcall needs.
>
> This will need to be backported wherever the commit above is backported
> (at least 2.5 for now).
>
I did not notice really those mem stats until now, good to know !
> For sure it's not perfect, and anything can always be improved. But
> that's not the point, the point here is to put info about the intent and
> the approach so that these ones can be preserved or corrected later. And
> that's valid for any patch, even a one-liner.
>
> The impact of your patch is that tcp-keep-alive will start to work on
> MacOS and if someone starts to rely on it and someone else later reverts
> your patch because it causes a build issue on a specific platform and the
> person thinks "oh we don't need this one there, we already have something
> else", by just reverting it they could break the other user's deployment.
>
> That's why it's important to put enough explanation to ensure the patch
> remains firmly stuck to the code base and cannot vanish without an extremly
> good argument.
>
Alrighty :-)
> Hoping this helps!
>
> Cheers
> Willy
[PATCH 1/1: BUILD/MINOR: TCP_KEEPIDLE macos equivalence
Hi here a little patch to set idle time for SO_KEEPALIVE socket option. Kind regards. 0001-BUILD-MINOR-socket-translate-TCP_KEEPIDLE-for-macOs-.patch Description: Binary data
Re: [PATCH 1/1]: BUILD/MEDIUM: tcp_sample porting get_tcp_info to macOs
Ok fair points. On Mon, 11 Apr 2022 at 06:38, Willy Tarreau wrote: > > Hi David, > > On Sat, Apr 09, 2022 at 02:44:54PM +0100, David CARLIER wrote: > > Hi, > > > > here a patch proposal to port a subset of the get_tcp_info samples to macOs. > > Thanks, that's quite interesting, but I really don't like the > reordering of the functions to try to group between OSes, for me this > is an indication that the granularity is not sufficient anymore. > > Instead I would suggest to put an explicit set of #if defined().../#endif > around each and every keyword, that will make it easier to spot which ones > are supported by what OS, and this will allow us to extend that as OSes > evolve. > > Please do that in two patches, one to split the ifdefs and a second > one to add MacOS support, this will also make it more obvious what info > is available on this platform. > > Thanks! > Willy 0001-BUILD-SMALL-tcp_sample-clarifying-samples-support-pe.patch Description: Binary data
Re: [PATCH 1/1]: BUILD/MEDIUM: tcp_sample porting get_tcp_info to macOs
On Mon, 11 Apr 2022 at 06:38, Willy Tarreau wrote: > > Hi David, > > On Sat, Apr 09, 2022 at 02:44:54PM +0100, David CARLIER wrote: > > Hi, > > > > here a patch proposal to port a subset of the get_tcp_info samples to macOs. > > Thanks, that's quite interesting, but I really don't like the > reordering of the functions to try to group between OSes, for me this > is an indication that the granularity is not sufficient anymore. > > Instead I would suggest to put an explicit set of #if defined().../#endif > around each and every keyword, that will make it easier to spot which ones > are supported by what OS, and this will allow us to extend that as OSes > evolve. > > Please do that in two patches, one to split the ifdefs and a second > one to add MacOS support, this will also make it more obvious what info > is available on this platform. > > Thanks! > Willy 0002-BUILD-MEDIUM-tcp_sample-porting-get_tcp_info-to-macO.patch Description: Binary data
[PATCH 1/1]: pool UAF pools ID setting on Linux 5.17 and onwards.
Hi,
Here a little patch for the use-after-free memory pool, allowing to
identify them.
Kind regards.
From 7f599d96c248ea6b7a67168ea68b31470c08b61d Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Sun, 10 Apr 2022 10:47:16 +0100
Subject: [PATCH] BUILD/MEDIUM: pool UAF's pool set an id on Linux 5.17 and
onwards.
With DEBUG_UAF DEBUG option, pools are created via anonymous pages
thus setting an ID upon them for debugging purpose.
---
src/pool.c | 25 -
1 file changed, 24 insertions(+), 1 deletion(-)
diff --git a/src/pool.c b/src/pool.c
index fe10badbd..d2c092243 100644
--- a/src/pool.c
+++ b/src/pool.c
@@ -13,6 +13,14 @@
#include
#include
+#if defined(__linux__)
+#include
+#if !defined(PR_SET_VMA)
+#define PR_SET_VMA 0x53564d41
+#define PR_SET_VMA_ANON_NAME 0
+#endif
+#endif
+
#include
#include
#include
@@ -770,10 +778,25 @@ void __pool_free(struct pool_head *pool, void *ptr)
void *pool_alloc_area_uaf(size_t size)
{
size_t pad = (4096 - size) & 0xFF0;
+ size_t tsize = (size + 4095) & -4096;
void *ret;
- ret = mmap(NULL, (size + 4095) & -4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
+ ret = mmap(NULL, tsize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
if (ret != MAP_FAILED) {
+#if defined(__linux__)
+ /* setting a id upon the whole page
+ * as long CONFIG_ANON_VMA_NAME is set (since 5.17)
+ * it copies (max 80 valid chars) in the kernel space
+ * so in /proc//maps the pools appear as
+ * `- rw-p 00:00 0 [anon:HAProxy debug UAF pool]`
+ * if it fails because of kernel version or the aforementioned config
+ * not set (like Fedora) it s inconsequential.
+ */
+ (void)prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME,
+(uintptr_t)ret,
+tsize,
+(uintptr_t)"HAProxy debug UAF pool");
+#endif
/* let's dereference the page before returning so that the real
* allocation in the system is performed without holding the lock.
*/
--
2.34.1
[PATCH 1/1]: BUILD/MEDIUM: tcp_sample porting get_tcp_info to macOs
Hi, here a patch proposal to port a subset of the get_tcp_info samples to macOs. Kind regards. 0001-BUILD-MEDIUM-tcp_sample-porting-get_tcp_info-and-sam.patch Description: Binary data
Re: [PATCH]: BUILD/MINOR: ssl openssl 3 warning fix
Alright fair enough will do. On Wed, 6 Apr 2022 at 10:06, William Lallemand wrote: > > On Wed, Apr 06, 2022 at 09:45:02AM +0100, David CARLIER wrote: > > > I recall there is a openssl3 port ongoing perhaps ? > > > > I was trying to see if the said 3.x portage work is close to be merged > > to master then yes my patch is useless. > > Otherwise > > In fact everything but the engine to provider port was merged! So you > should only have warning about that. > > We will probably keep the engine support in the code and disabled it by > default. But what could be done is adding > -DOPENSSL_API_COMPAT=0x1010L to the compiler command when activating > USE_ENGINE. > > > > > but here a little "band-aid" proposal for the actual master branch. > > > > my patch is just a temporary fix. > > > > At the moment we prefer to keep track of the warning because we need to > fix them properly, but on the CI we are actually doing: > make DEFINE="-DOPENSSL_API_COMPAT=0x1010L -DOPENSSL_NO_DEPRECATED" > > > > Regards, > > -- > William Lallemand
Re: [PATCH]: BUILD/MINOR: ssl openssl 3 warning fix
On Wed, 6 Apr 2022 at 09:37, William Lallemand wrote: > > Hello David, > > On Tue, Apr 05, 2022 at 07:07:47PM +0100, David CARLIER wrote: > > Hi, I recall there is a openssl3 port ongoing perhaps ? > > > > but here a little "band-aid" proposal for the actual master branch. > > > > From 2d76cb9f249b9519d2b102133b224716965f3f02 Mon Sep 17 00:00:00 2001 > > From: David Carlier > > Date: Tue, 5 Apr 2022 19:03:05 +0100 > > Subject: [PATCH] BUID/MINOR: ssl fix warning build with openssl 3 > > > > fix warning build from various deprecated apis from the 3.x release. > > I'm not sure what deprecated warning you are seeing, I only have the one about > the Engine API, that will be disabled by default later. Fair enough by my old question > I recall there is a openssl3 port ongoing perhaps ? I was trying to see if the said 3.x portage work is close to be merged to master then yes my patch is useless. Otherwise > but here a little "band-aid" proposal for the actual master branch. my patch is just a temporary fix. > > > --- > > include/haproxy/openssl-compat.h | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/include/haproxy/openssl-compat.h > > b/include/haproxy/openssl-compat.h > > index 87bb5109c..8a252aef9 100644 > > --- a/include/haproxy/openssl-compat.h > > +++ b/include/haproxy/openssl-compat.h > > @@ -2,6 +2,7 @@ > > #define _HAPROXY_OPENSSL_COMPAT_H > > #ifdef USE_OPENSSL > > > > +#define OPENSSL_API_COMPAT 0x1010L > > #include > > #include > > #include > > -- > > 2.34.1 > > That is not a good idea in my opinion, the goal is a real portage to the 3.0 > API, once it's done it is supposed to compile with OPENSSL_NO_DEPRECATED > defined. > > -- > William Lallemand
[PATCH]: BUILD/MINOR: ssl openssl 3 warning fix
Hi, I recall there is a openssl3 port ongoing perhaps ? but here a little "band-aid" proposal for the actual master branch. From 2d76cb9f249b9519d2b102133b224716965f3f02 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Tue, 5 Apr 2022 19:03:05 +0100 Subject: [PATCH] BUID/MINOR: ssl fix warning build with openssl 3 fix warning build from various deprecated apis from the 3.x release. --- include/haproxy/openssl-compat.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/haproxy/openssl-compat.h b/include/haproxy/openssl-compat.h index 87bb5109c..8a252aef9 100644 --- a/include/haproxy/openssl-compat.h +++ b/include/haproxy/openssl-compat.h @@ -2,6 +2,7 @@ #define _HAPROXY_OPENSSL_COMPAT_H #ifdef USE_OPENSSL +#define OPENSSL_API_COMPAT 0x1010L #include #include #include -- 2.34.1
[PATCH] BUILD/MEDIUM: FreeBSD < 14 build fix
Hi to (re)close #1555. :-) cheers. From afc36e0ee99504f4da82815256a374179872b148 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Tue, 8 Mar 2022 14:49:29 + Subject: [PATCH] BUILD/MEDIUM: freebsd build fix. Supporting kFreebsd previously led to FreeBSD (< 14) build breakage. --- include/haproxy/cpuset-t.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/haproxy/cpuset-t.h b/include/haproxy/cpuset-t.h index 984df8d83..b770b1791 100644 --- a/include/haproxy/cpuset-t.h +++ b/include/haproxy/cpuset-t.h @@ -16,7 +16,8 @@ #include -#if defined(__linux__) || defined(__DragonFly__) || defined(__FreeBSD_kernel__) +#if defined(__linux__) || defined(__DragonFly__) || \ + (defined(__FreeBSD_kernel__) && defined(__GLIBC__)) # define CPUSET_REPR cpu_set_t # define CPUSET_USE_CPUSET -- 2.35.1
[PATCH] BUILD/MEDIUM kFreeBSD build fix
Hi,
here the patch related to the GH issue #1555.
Kind regards.
From 02bd5ee5089c20a99ad3dfba433dd0e3eda2ceee Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Fri, 4 Mar 2022 15:50:48 +
Subject: [PATCH] BUILD/MEDIUM: fix kFreeBSD build.
kFreeBSD needs to be treated as a distinct target from FreeBSD
since the underlying system libc is the GNU one. Thus, relying
only __GLIBC__ no longer suffice.
- freebsd-glibc new target, key difference is including crypt.h
and linking to libdl like linux.
- cpu affinity available but the api is still the FreeBSD's.
- enabling auxiliary data access only for Linux.
Patch based on preliminary work done by @bigon.
closes #1555
---
Makefile | 11 +--
include/haproxy/cpuset-t.h | 2 +-
src/tools.c| 4 ++--
3 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/Makefile b/Makefile
index 964b7fbb5..46da6fbbb 100644
--- a/Makefile
+++ b/Makefile
@@ -177,8 +177,8 @@ DOCDIR = $(PREFIX)/doc/haproxy
TARGET system
# Use TARGET= to optimize for a specific target OS among the
# following list (use the default "generic" if uncertain) :
-#linux-glibc, linux-glibc-legacy, linux-musl, solaris, freebsd, dragonfly,
-#openbsd, netbsd, cygwin, haiku, aix51, aix52, aix72-gcc, osx, generic,
+#linux-glibc, linux-glibc-legacy, linux-musl, solaris, freebsd, freebsd-glibc,
+#dragonfly, openbsd, netbsd, cygwin, haiku, aix51, aix52, aix72-gcc, osx, generic,
#custom
TARGET =
@@ -412,6 +412,13 @@ ifeq ($(TARGET),freebsd)
USE_ACCEPT4 USE_CLOSEFROM USE_GETADDRINFO USE_PROCCTL)
endif
+# kFreeBSD glibc
+ifeq ($(TARGET),freebsd-glibc)
+ set_target_defaults = $(call default_opts, \
+USE_POLL USE_TPROXY USE_LIBCRYPT USE_THREAD USE_CPU_AFFINITY USE_KQUEUE \
+USE_ACCEPT4 USE_GETADDRINFO USE_CRYPT_H USE_DL)
+endif
+
# DragonFlyBSD 4.3 and above
ifeq ($(TARGET),dragonfly)
set_target_defaults = $(call default_opts, \
diff --git a/include/haproxy/cpuset-t.h b/include/haproxy/cpuset-t.h
index 5f812aa17..984df8d83 100644
--- a/include/haproxy/cpuset-t.h
+++ b/include/haproxy/cpuset-t.h
@@ -16,7 +16,7 @@
#include
-#if defined(__linux__) || defined(__DragonFly__)
+#if defined(__linux__) || defined(__DragonFly__) || defined(__FreeBSD_kernel__)
# define CPUSET_REPR cpu_set_t
# define CPUSET_USE_CPUSET
diff --git a/src/tools.c b/src/tools.c
index c48a9698c..f62032b9c 100644
--- a/src/tools.c
+++ b/src/tools.c
@@ -43,7 +43,7 @@ extern void *__elf_aux_vector;
#include
#include
-#if defined(__GLIBC__) && (__GLIBC__ > 2 || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 16))
+#if defined(__linux__) && defined(__GLIBC__) && (__GLIBC__ > 2 || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 16))
#include
#endif
@@ -4791,7 +4791,7 @@ const char *get_exec_path()
{
const char *ret = NULL;
-#if defined(__GLIBC__) && (__GLIBC__ > 2 || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 16))
+#if defined(__linux__) && defined(__GLIBC__) && (__GLIBC__ > 2 || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 16))
long execfn = getauxval(AT_EXECFN);
if (execfn && execfn != ENOENT)
--
2.34.1
Re: [EXTERNAL] Re: [PATCH]: thread basic solaris support
understandable. cheers :) On Mon, 14 Feb 2022 at 08:20, Willy TARREAU wrote: > > On Mon, Feb 14, 2022 at 08:11:14AM +, David CARLIER wrote: > > ping :) > > Sorry David, my tree is currently full of debugging code and functions > with guts hanging out, which is why I tend to postpone merging of non- > critical stuff. I'll take care of it soon though. > > Thanks, > Willy
Re: [PATCH]: thread basic solaris support
ping :) On Mon, 31 Jan 2022 at 19:37, David CARLIER wrote: > > Hi, > > here a patch proposal for a solaris cpu affinity basic support (no NUMA > (yet)). > > Kind regards.
[PATCH]: thread basic solaris support
Hi,
here a patch proposal for a solaris cpu affinity basic support (no NUMA (yet)).
Kind regards.
From e3afd3de4f8c4e1d4d0d1d98a8de814aed19ffa6 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Mon, 31 Jan 2022 19:31:53 +
Subject: [PATCH] MEDIUM: thread basic cpu affinity support for solaris based
systems.
focusing on being able to attribute threads to cpu for now (cpu locality
not yet sure if possible w/o too convoluted code).
Using here native API from both solaris and illumos.
---
Makefile | 2 +-
include/haproxy/cpuset-t.h | 5 +
src/cpuset.c | 19 ++-
src/thread.c | 13 -
4 files changed, 32 insertions(+), 7 deletions(-)
diff --git a/Makefile b/Makefile
index 57c7510a1..88acc7ac8 100644
--- a/Makefile
+++ b/Makefile
@@ -377,7 +377,7 @@ endif
ifeq ($(TARGET),solaris)
set_target_defaults = $(call default_opts, \
USE_POLL USE_TPROXY USE_LIBCRYPT USE_CRYPT_H USE_GETADDRINFO USE_THREAD \
-USE_RT USE_OBSOLETE_LINKER USE_EVPORTS USE_CLOSEFROM)
+USE_RT USE_OBSOLETE_LINKER USE_EVPORTS USE_CLOSEFROM USE_CPU_AFFINITY)
TARGET_CFLAGS = -DFD_SETSIZE=65536 -D_REENTRANT -D_XOPEN_SOURCE=600 -D__EXTENSIONS__
TARGET_LDFLAGS = -lnsl -lsocket
endif
diff --git a/include/haproxy/cpuset-t.h b/include/haproxy/cpuset-t.h
index 5f812aa17..e362d9620 100644
--- a/include/haproxy/cpuset-t.h
+++ b/include/haproxy/cpuset-t.h
@@ -36,6 +36,11 @@
# define CPUSET_REPR unsigned long
# define CPUSET_USE_ULONG
+#elif defined(__sun)
+
+# define CPUSET_REPR int
+# define CPUSET_USE_PSET
+
#else
# error "No cpuset support implemented on this platform"
diff --git a/src/cpuset.c b/src/cpuset.c
index f7b66020e..a8aa6995d 100644
--- a/src/cpuset.c
+++ b/src/cpuset.c
@@ -12,7 +12,7 @@ void ha_cpuset_zero(struct hap_cpuset *set)
#if defined(CPUSET_USE_CPUSET) || defined(CPUSET_USE_FREEBSD_CPUSET)
CPU_ZERO(>cpuset);
-#elif defined(CPUSET_USE_ULONG)
+#elif defined(CPUSET_USE_ULONG) || defined(CPUSET_USE_PSET)
set->cpuset = 0;
#endif
}
@@ -26,7 +26,7 @@ int ha_cpuset_set(struct hap_cpuset *set, int cpu)
CPU_SET(cpu, >cpuset);
return 0;
-#elif defined(CPUSET_USE_ULONG)
+#elif defined(CPUSET_USE_ULONG) || defined(CPUSET_USE_PSET)
set->cpuset |= (0x1 << cpu);
return 0;
#endif
@@ -41,7 +41,7 @@ int ha_cpuset_clr(struct hap_cpuset *set, int cpu)
CPU_CLR(cpu, >cpuset);
return 0;
-#elif defined(CPUSET_USE_ULONG)
+#elif defined(CPUSET_USE_ULONG) || defined(CPUSET_USE_PSET)
set->cpuset &= ~(0x1 << cpu);
return 0;
#endif
@@ -55,7 +55,7 @@ void ha_cpuset_and(struct hap_cpuset *dst, struct hap_cpuset *src)
#elif defined(CPUSET_USE_FREEBSD_CPUSET)
CPU_AND(>cpuset, >cpuset);
-#elif defined(CPUSET_USE_ULONG)
+#elif defined(CPUSET_USE_ULONG) || defined(CPUSET_USE_PSET)
dst->cpuset &= src->cpuset;
#endif
}
@@ -67,6 +67,8 @@ int ha_cpuset_count(const struct hap_cpuset *set)
#elif defined(CPUSET_USE_ULONG)
return my_popcountl(set->cpuset);
+#elif defined(CPUSET_USE_PSET)
+ return __builtin_popcount(set->cpuset);
#endif
}
@@ -91,6 +93,11 @@ int ha_cpuset_ffs(const struct hap_cpuset *set)
return 0;
return my_ffsl(set->cpuset);
+#elif defined(CPUSET_USE_PSET)
+ if (!set->cpuset)
+ return 0;
+
+ return __builtin_ffs(set->cpuset);
#endif
}
@@ -103,7 +110,7 @@ void ha_cpuset_assign(struct hap_cpuset *dst, struct hap_cpuset *src)
#elif defined(CPUSET_USE_FREEBSD_CPUSET)
CPU_COPY(>cpuset, >cpuset);
-#elif defined(CPUSET_USE_ULONG)
+#elif defined(CPUSET_USE_ULONG) || defined(CPUSET_USE_PSET)
dst->cpuset = src->cpuset;
#endif
}
@@ -115,6 +122,8 @@ int ha_cpuset_size()
#elif defined(CPUSET_USE_ULONG)
return LONGBITS;
+#elif defined(CPUSET_USE_PSET)
+ return sizeof(int) * 8;
#endif
}
diff --git a/src/thread.c b/src/thread.c
index a04e914d9..87295f2c9 100644
--- a/src/thread.c
+++ b/src/thread.c
@@ -39,6 +39,9 @@
#include
#include
# endif
+# ifdef __sun
+#include
+# endif
# include
#endif
@@ -250,17 +253,25 @@ void set_thread_cpu_affinity()
ha_cpuset_and(_map.thread[tid], _map.proc);
if (ha_cpuset_count(_map.thread[tid])) {/* only do this if the thread has a THREAD map */
-# if defined(__APPLE__)
+# if defined(__APPLE__) || defined(__sun)
/* Note: this API is limited to the first 32/64 CPUs */
unsigned long set = cpu_map.thread[tid].cpuset;
int j;
while ((j = ffsl(set)) > 0) {
+#if defined(__APPLE__)
thread_affinity_policy_data_t cpu_set = { j - 1 };
thread_port_t mthread;
mthread = pthread_mach_thread_np(ha_pthread[tid]);
thread_policy_set(mthread, THREAD_AFFINITY_POLICY, (thread_policy_t)_set, 1);
+#else
+ psetid_t cpu_set;
+ pset_create(_set);
+ pset_assign(cpu_set, j - 1, NULL);
+ pset_bind(cpu_set, P_PID, getpid(), NULL);
+ pset_destroy(cpu_set);
+#endif
set &= ~(1UL << (j - 1));
}
# else
--
2.35.1
Re: [PATCH 3/3]: deviceatlas update doc and build changes.
Here find attached corrected versions. Thanks. Regards. On Thu, Jan 27, 2022 at 5:37 PM Willy Tarreau wrote: > On Fri, Jan 21, 2022 at 09:05:18PM +0000, David Carlier wrote: > > From e3f520ebd4f23fe42a1068ee2d9b42fd529241e8 Mon Sep 17 00:00:00 2001 > > From: David Carlier > > Date: Fri, 21 Jan 2022 20:57:27 + > > Subject: [PATCH 3/3] MEDIUM: deviceatlas doc and build update. > > "MEDIUM: da: doc and build for new runtime update service" ? > > > diff --git a/Makefile b/Makefile > > index af0f5cc87..e6f2e04ad 100644 > > --- a/Makefile > > +++ b/Makefile > > @@ -653,11 +653,14 @@ endif > > ifneq ($(USE_PCRE2),) > > OPTIONS_CFLAGS += -DDA_REGEX_HDR=\"dac_pcre2.c\" -DDA_REGEX_TAG=2 > > endif > > +OPTIONS_OBJS += $(DEVICEATLAS_LIB)/Os/daunix.o > > +OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dadwcom.o > > +OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dasch.o > > OPTIONS_OBJS += $(DEVICEATLAS_LIB)/json.o > > OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dac.o > > endif > > OPTIONS_OBJS += addons/deviceatlas/da.o > > -OPTIONS_CFLAGS += $(if $(DEVICEATLAS_INC),-I$(DEVICEATLAS_INC)) > > +OPTIONS_CFLAGS += $(if $(DEVICEATLAS_INC),-I$(DEVICEATLAS_INC)) > ^ > > Unneeded change that only inserted an accidental space. > > > --- a/doc/DeviceAtlas-device-detection.txt > > +++ b/doc/DeviceAtlas-device-detection.txt > > @@ -9,7 +9,8 @@ The build supports the USE_PCRE and USE_PCRE2 options. > Once extracted : > > > > Optionally DEVICEATLAS_INC and DEVICEATLAS_LIB may be set to override > the path > > to the include files and libraries respectively if they're not in the > source > > -directory. > > +directory. However, if the API had been installed beforehand, > DEVICEATLAS_SRC > > +can be omitted. Note that the DeviceAtlas C API version supported is > the 2.4.0 at minimum. > > Please trim your text lines a bit, the rest of the file is justified to 80 > chars > except for copy-pastable commands. The one above and a few below become > really > hard to read in a terminal while previously that file managed to remain > pretty > clean. There are a few other such lines that got extended further in the > patch. > > However now when trying to build I'm getting this new error: > > make: *** No rule to make target 'addons/deviceatlas/dummy/Os/daunix.o', > needed by 'haproxy'. Stop. > > Similarly I think you're missing an update to the dummy lib so that it's > transparent for build test and basic testing (i.e. I don't think you need > to > implement anything looking like a real update, you can claim it happens > every > few calls for example or even never, whatever suits you best). > > Thank you! > Willy > 0003-MEDIUM-da-update-doc-and-build-for-new-scheduler-mod.patch Description: Binary data
Re: [PATCH 2/3]: deviceatlas addon update
On Thu, Jan 27, 2022 at 5:29 PM Willy Tarreau wrote:
> On Fri, Jan 21, 2022 at 09:04:25PM +0000, David Carlier wrote:
> > Second patch of the patchset concerning the update of the addon itself.
>
> > From d541d8dfd8aa0290625b3824bd6b44d2861e997f Mon Sep 17 00:00:00 2001
> > From: David Carlier
> > Date: Fri, 21 Jan 2022 20:51:20 +
> > Subject: [PATCH 2/3] MEDIUM: deviceatlas update of the main module.
>^^^
>da:
>
> Also please avoid generic messages like "update the main module" because
> an update is not the purpose of a commit, which by definition is an update.
> Prefer more explanattory messages such as:
>
> "MEDIUM: da: support database runtime updates"
>
> > The DeviceAtlas addon can optionally interacts with the new service
> > without change of configuration in the HAProxy part.
> > Note that however it requires the DeviceAtlas Identification C API
> > 2.4.0 minimum from this point.
> >
> > Signed-off-by: David Carlier
>
> (...)
> > +#ifdef USE_THREAD
> > +__decl_spinlock(dadwsch_lock);
> > +#endif
>
> For this one you can do that:
>
> __decl_thread(HA_SPINLOCK_T dadwsch_lock);
>
> It will only do it when USE_THREAD is set and save you an ifdef.
>
> > @@ -163,6 +177,16 @@ static int init_deviceatlas(void)
> >
> > global_deviceatlas.useragentid =
> da_atlas_header_evidence_id(_deviceatlas.atlas,
> > "user-agent");
> > + if ((global_deviceatlas.atlasfd = shm_open(ATLASMAPNM,
> O_RDWR, 0660)) != -1) {
> > + global_deviceatlas.atlasmap = mmap(NULL,
> ATLASTOKSZ, PROT_READ | PROT_WRITE, MAP_SHARED, global_deviceatlas.atlasfd,
> 0);
> > + if (global_deviceatlas.atlasmap == MAP_FAILED) {
> > + close(global_deviceatlas.atlasfd);
> > + global_deviceatlas.atlasfd = -1;
> > + global_deviceatlas.atlasmap = NULL;
> > + } else {
> > + fprintf(stdout, "deviceatlas : scheduling
> support.\n");
>
> This message is still confusing because it's unclear whether it reports
> a detected feature or an anomaly. Think for example about someone facing
> it on a machine exhibiting a problem and not having it on a machine that
> works fine, it would then be understandable that it's taken for an error
> message. Maybe "Deviceatlas: scheduling support enabled." would do it ?
>
> OK for the rest of this patch at first glance, however now I'm getting
> this build error:
>
> addons/deviceatlas/da.c: In function 'da_haproxy_checkinst':
> addons/deviceatlas/da.c:236:8: warning: implicit declaration of function
> 'da_atlas_read_mapped' [-Wimplicit-function-declaration]
> addons/deviceatlas/da.o: In function `da_haproxy_checkinst':
> /g/public/haproxy/addons/deviceatlas/da.c:236: undefined reference to
> `da_atlas_read_mapped'
> collect2: error: ld returned 1 exit status
> distcc[12485] ERROR: compile (null) on localhost failed
> make: *** [Makefile:952: haproxy] Error 1
>
> So it seems to me that the dummy lib missed an update (I guess you tested
> with the real lib only and forgot about that one).
>
> Willy
>
0002-MEDIUM-da-module-to-handle-schedule-mode.patch
Description: Binary data
Re: [PATCH 1/3] deviceatlas update new service addition
Hi willy and thanks for the review. Indeed I forgot about the dummy part, changes present in the last patch. On Thu, Jan 27, 2022 at 5:18 PM Willy Tarreau wrote: > Hi David, > > finally back to your patches. There are very minor things but I'm not > going to modify them without your consent since they're signed-off. > > > From eae56a44d3bf51f8e92f66baef25f2a417f837ce Mon Sep 17 00:00:00 2001 > > From: David Carlier > > Date: Fri, 21 Jan 2022 20:46:40 + > > Subject: [PATCH 1/3] MEDIUM: deviceatlas new optional data file > download scheduler service. > ^^^ > Please always place a colon after the subsystem. Also, previously it > was referred to as "da", it would be better to stay consistent in order > to more easily list all such related changes. > > > New specialized service to daily handle the update of download file > without > > interruption of service and to be preemptively started before HAProxy. > > While I do know from your private explanation that it's an external > service, it's not immediately obvious to the reader. Here it would > be welcome to mention that it's a standalone utility that accesses > a shared memory used by the DA module and explain a little bit how > it works (the general principles and choices). And please mention it > uses libcurl since it's a new dependency, this may save some trial > and errors to those in the process of a bisect for example. > > Also, I noticed that it doesn't build with the dummy lib. Since it's > a standalone tool I don't think it's a problem, but this really needs > to be mentioned in the commit message, ideally with a reminder about > what's missing (either a link to the download site or an invitation > to read build instructions from doc/DeviceAtlas-device-detection.txt). > > > Signed-off-by: David Carlier > > --- > > addons/deviceatlas/Makefile | 49 + > > addons/deviceatlas/dadwsch.c | 195 +++ > > 2 files changed, 244 insertions(+) > > create mode 100644 addons/deviceatlas/Makefile > > create mode 100644 addons/deviceatlas/dadwsch.c > (...) > > When I applied it I got these warnings: > > $ git am 0001-MEDIUM-deviceatlas-new-optional-data-file-download-s.patch > Applying: MEDIUM: deviceatlas new optional data file download scheduler > service. > .git/rebase-apply/patch:14: trailing whitespace. > # DEVICEATLAS_SRC : DeviceAtlas API source root path > .git/rebase-apply/patch:62: new blank line at EOF. > + > warning: 2 lines add whitespace errors. > > One of them is at the end of the line below: > > > diff --git a/addons/deviceatlas/Makefile b/addons/deviceatlas/Makefile > > new file mode 100644 > > index 0..33dd9434e > > --- /dev/null > > +++ b/addons/deviceatlas/Makefile > > @@ -0,0 +1,49 @@ > > +# DEVICEATLAS_SRC : DeviceAtlas API source root path >^ here > > An easy and convenient way to catch them before committing (or see them > after to ease their fixing) is to add this into your $HOME/.gitconfig: > > [color] >ui = true > > A "git diff" or "git show" will show them on a pretty visible red > background, > that's how most of us avoid and fix them.. > > The second one is the empty line at the end of the Makefile. > > Aside these details it's OK for me. > > Willy > 0001-MEDIUM-da-new-optional-data-file-download-scheduler-.patch Description: Binary data
Re: [PATCH] BUILD/MEDIUM: debug haiku build fix
Hi, sure whatever solution you deem better. On Tue, 25 Jan 2022 at 11:09, Willy Tarreau wrote: > > Hi David, > > On Tue, Jan 25, 2022 at 10:40:57AM +, David CARLIER wrote: > > From 575cdb49b8fa923bdd44fe08a7068681e38ffe71 Mon Sep 17 00:00:00 2001 > > From: David Carlier > > Date: Tue, 25 Jan 2022 10:37:59 + > > Subject: [PATCH] BUILD/MINOR: debug haiku build fix. > > > > O_ASYNC fcntl flag equivalent on this platform is O_RSYNC. > > --- > > src/debug.c | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/src/debug.c b/src/debug.c > > index c15c92650..bd6353846 100644 > > --- a/src/debug.c > > +++ b/src/debug.c > > @@ -25,6 +25,9 @@ > > #ifdef USE_EPOLL > > #include > > #endif > > +#ifdef __HAIKU__ > > +#define O_ASYNC O_RSYNC > > +#endif > > Hmmm I see, we didn't use O_ASYNC previously. However I'm surprised with > the use of this flag, usually O_ASYNC tends to be remapped to O_NONBLOCK > (that we already use everywhere else). As such I would have done a possibly > more portable: > > #ifndef O_ASYNC > #define O_ASYNC O_NONBLOCK > #endif > > I could/should do even better in fact, which consists in conditioning > such flags to their definition as is already done for a few of them. > > Willy
[PATCH] BUILD/MEDIUM: debug haiku build fix
Hi, here a little patch for haiku system. cheers. From 575cdb49b8fa923bdd44fe08a7068681e38ffe71 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Tue, 25 Jan 2022 10:37:59 + Subject: [PATCH] BUILD/MINOR: debug haiku build fix. O_ASYNC fcntl flag equivalent on this platform is O_RSYNC. --- src/debug.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/debug.c b/src/debug.c index c15c92650..bd6353846 100644 --- a/src/debug.c +++ b/src/debug.c @@ -25,6 +25,9 @@ #ifdef USE_EPOLL #include #endif +#ifdef __HAIKU__ +#define O_ASYNC O_RSYNC +#endif #include #include -- 2.30.2
[PATCH 3/3]: deviceatlas update doc and build changes.
Last patch of the patchset concerning the doc and the build. Thanks in advance. From e3f520ebd4f23fe42a1068ee2d9b42fd529241e8 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Fri, 21 Jan 2022 20:57:27 + Subject: [PATCH 3/3] MEDIUM: deviceatlas doc and build update. Mentions of the new service and update of the build. --- Makefile | 5 - doc/DeviceAtlas-device-detection.txt | 14 +- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index af0f5cc87..e6f2e04ad 100644 --- a/Makefile +++ b/Makefile @@ -653,11 +653,14 @@ endif ifneq ($(USE_PCRE2),) OPTIONS_CFLAGS += -DDA_REGEX_HDR=\"dac_pcre2.c\" -DDA_REGEX_TAG=2 endif +OPTIONS_OBJS += $(DEVICEATLAS_LIB)/Os/daunix.o +OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dadwcom.o +OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dasch.o OPTIONS_OBJS += $(DEVICEATLAS_LIB)/json.o OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dac.o endif OPTIONS_OBJS += addons/deviceatlas/da.o -OPTIONS_CFLAGS += $(if $(DEVICEATLAS_INC),-I$(DEVICEATLAS_INC)) +OPTIONS_CFLAGS += $(if $(DEVICEATLAS_INC),-I$(DEVICEATLAS_INC)) endif ifneq ($(USE_51DEGREES),) diff --git a/doc/DeviceAtlas-device-detection.txt b/doc/DeviceAtlas-device-detection.txt index 9a1c9b594..0045448f5 100644 --- a/doc/DeviceAtlas-device-detection.txt +++ b/doc/DeviceAtlas-device-detection.txt @@ -9,7 +9,8 @@ The build supports the USE_PCRE and USE_PCRE2 options. Once extracted : Optionally DEVICEATLAS_INC and DEVICEATLAS_LIB may be set to override the path to the include files and libraries respectively if they're not in the source -directory. +directory. However, if the API had been installed beforehand, DEVICEATLAS_SRC +can be omitted. Note that the DeviceAtlas C API version supported is the 2.4.0 at minimum. For HAProxy developers who need to verify that their changes didn't accidentally break the DeviceAtlas code, it is possible to build a dummy library provided in @@ -62,5 +63,16 @@ Single HTTP header acl device_type_tablet req.fhdr(User-Agent),da-csv-conv(primaryHardwareType) "Tablet" +Optionally a JSON download scheduler is provided to allow a data file being fetched automatically in a daily basis without +restarting HAProxy : + +$ cd addons/deviceatlas && make [DEVICEATLAS_SRC=] + +Similarly, if the DeviceAtlas API is installed, DEVICEATLAS_SRC can be omitted. + +$ ./dadwsch -u JSON data file URL e.g. "https://deviceatlas.com/getJSON?licencekey==zip=my=web" [-p download directory path /tmp by default] [-d scheduled hour of download, hour when the service is launched by default] + +Noted it needs to be started before HAProxy. + Please find more information about DeviceAtlas and the detection methods at https://deviceatlas.com/resources . -- 2.30.2
[PATCH 2/3]: deviceatlas addon update
Second patch of the patchset concerning the update of the addon itself.
From d541d8dfd8aa0290625b3824bd6b44d2861e997f Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Fri, 21 Jan 2022 20:51:20 +
Subject: [PATCH 2/3] MEDIUM: deviceatlas update of the main module.
The DeviceAtlas addon can optionally interacts with the new service
without change of configuration in the HAProxy part.
Note that however it requires the DeviceAtlas Identification C API
2.4.0 minimum from this point.
Signed-off-by: David Carlier
---
addons/deviceatlas/da.c | 77 +
1 file changed, 77 insertions(+)
diff --git a/addons/deviceatlas/da.c b/addons/deviceatlas/da.c
index 418909226..724209936 100644
--- a/addons/deviceatlas/da.c
+++ b/addons/deviceatlas/da.c
@@ -1,4 +1,7 @@
#include
+#include
+#include
+#include
#include
#include
@@ -14,11 +17,16 @@
#include
#include
+#define ATLASTOKSZ PATH_MAX
+#define ATLASMAPNM "/hapdeviceatlas"
+
static struct {
void *atlasimgptr;
+ void *atlasmap;
char *jsonpath;
char *cookiename;
size_t cookienamelen;
+ int atlasfd;
da_atlas_t atlas;
da_evidence_id_t useragentid;
da_severity_t loglevel;
@@ -29,11 +37,17 @@ static struct {
.jsonpath = 0,
.cookiename = 0,
.cookienamelen = 0,
+ .atlasmap = NULL,
+ .atlasfd = -1,
.useragentid = 0,
.daset = 0,
.separator = '|',
};
+#ifdef USE_THREAD
+__decl_spinlock(dadwsch_lock);
+#endif
+
static int da_json_file(char **args, int section_type, struct proxy *curpx,
const struct proxy *defpx, const char *file, int line,
char **err)
@@ -163,6 +177,16 @@ static int init_deviceatlas(void)
global_deviceatlas.useragentid = da_atlas_header_evidence_id(_deviceatlas.atlas,
"user-agent");
+ if ((global_deviceatlas.atlasfd = shm_open(ATLASMAPNM, O_RDWR, 0660)) != -1) {
+ global_deviceatlas.atlasmap = mmap(NULL, ATLASTOKSZ, PROT_READ | PROT_WRITE, MAP_SHARED, global_deviceatlas.atlasfd, 0);
+ if (global_deviceatlas.atlasmap == MAP_FAILED) {
+close(global_deviceatlas.atlasfd);
+global_deviceatlas.atlasfd = -1;
+global_deviceatlas.atlasmap = NULL;
+ } else {
+fprintf(stdout, "deviceatlas : scheduling support.\n");
+ }
+ }
global_deviceatlas.daset = 1;
fprintf(stdout, "Deviceatlas module loaded.\n");
@@ -184,9 +208,58 @@ static void deinit_deviceatlas(void)
free(global_deviceatlas.atlasimgptr);
}
+ if (global_deviceatlas.atlasfd != -1) {
+ munmap(global_deviceatlas.atlasmap, ATLASTOKSZ);
+ close(global_deviceatlas.atlasfd);
+ shm_unlink(ATLASMAPNM);
+ }
+
da_fini();
}
+static void da_haproxy_checkinst(void)
+{
+ if (global_deviceatlas.atlasmap != 0) {
+ char *base;
+ base = (char *)global_deviceatlas.atlasmap;
+
+ if (base[0] != 0) {
+ void *cnew;
+ size_t atlassz;
+ char atlasp[ATLASTOKSZ] = {0};
+ da_atlas_t inst;
+ da_property_decl_t extraprops[1] = {{NULL, 0}};
+#ifdef USE_THREAD
+ HA_SPIN_LOCK(OTHER_LOCK, _lock);
+#endif
+ strlcpy2(atlasp, base, sizeof(atlasp));
+ if (da_atlas_read_mapped(atlasp, NULL, , ) == DA_OK) {
+if (da_atlas_open(, extraprops, cnew, atlassz) == DA_OK) {
+ char jsonbuf[26];
+ time_t jsond;
+
+ da_atlas_close(_deviceatlas.atlas);
+ free(global_deviceatlas.atlasimgptr);
+ global_deviceatlas.atlasimgptr = cnew;
+ global_deviceatlas.atlas = inst;
+ memset(base, 0, ATLASTOKSZ);
+ jsond = da_getdatacreation(_deviceatlas.atlas);
+ ctime_r(, jsonbuf);
+ jsonbuf[24] = 0;
+ printf("deviceatlas: new instance, data file date `%s`.\n", jsonbuf);
+} else {
+ ha_warning("deviceatlas: instance update failed.\n");
+ memset(base, 0, ATLASTOKSZ);
+ free(cnew);
+}
+ }
+#ifdef USE_THREAD
+ HA_SPIN_UNLOCK(OTHER_LOCK, _lock);
+#endif
+ }
+ }
+}
+
static int da_haproxy(const struct arg *args, struct sample *smp, da_deviceinfo_t *devinfo)
{
struct buffer *tmp;
@@ -272,6 +345,8 @@ static int da_haproxy_conv(const struct arg *args, struct sample *smp, void *pri
return 1;
}
+ da_haproxy_checkinst();
+
i = smp->data.u.str.data > sizeof(useragentbuf) ? sizeof(useragentbuf) : smp->data.u.str.data;
memcpy(useragentbuf, smp->data.u.str.area, i - 1);
useragentbuf[i - 1] = 0;
@@ -301,6 +376,8 @@ static int da_haproxy_fetch(const struct arg *args, struct sample *smp, const ch
return 0;
}
+ da_haproxy_checkinst();
+
chn = (smp->strm ? >strm->req : NULL);
htx = smp_prefetch_htx(smp, chn, NULL, 1);
if (!htx)
--
2.30.2
[PATCH 1/3] deviceatlas update new service addition
Hi here the first patch of the patchset for the DeviceAtlas addon
concerning the
new download/scheduler service.
Thanks in advance.
From eae56a44d3bf51f8e92f66baef25f2a417f837ce Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Fri, 21 Jan 2022 20:46:40 +
Subject: [PATCH 1/3] MEDIUM: deviceatlas new optional data file download
scheduler service.
New specialized service to daily handle the update of download file without
interruption of service and to be preemptively started before HAProxy.
Signed-off-by: David Carlier
---
addons/deviceatlas/Makefile | 49 +
addons/deviceatlas/dadwsch.c | 195 +++
2 files changed, 244 insertions(+)
create mode 100644 addons/deviceatlas/Makefile
create mode 100644 addons/deviceatlas/dadwsch.c
diff --git a/addons/deviceatlas/Makefile b/addons/deviceatlas/Makefile
new file mode 100644
index 0..33dd9434e
--- /dev/null
+++ b/addons/deviceatlas/Makefile
@@ -0,0 +1,49 @@
+# DEVICEATLAS_SRC : DeviceAtlas API source root path
+
+
+OS := $(shell uname -s)
+OBJS:= dadwsch.o
+CFLAGS := -g -O2
+LDFLAGS :=
+
+CURL_CONFIG := curl-config
+CURLDIR := $(shell $(CURL_CONFIG) --prefix 2>/dev/null || echo /usr/local)
+CURL_INC:= $(CURLDIR)/include
+CURL_LIB:= $(CURLDIR)/lib
+CURL_LDFLAGS:= $(shell $(CURL_CONFIG) --libs 2>/dev/null || echo -L /usr/local/lib -lcurl)
+
+PCRE2_CONFIG:= pcre2-config
+PCRE2DIR:= $(shell $(PCRE2_CONFIG) --prefix 2>/dev/null || echo /usr/local)
+PCRE2_INC := $(PCRE2DIR)/include
+PCRE2_LIB := $(PCRE2DIR)/lib
+PCRE2_LDFLAGS := $(shell $(PCRE2_CONFIG) --libs8 2>/dev/null || echo /usr/local)
+
+ifeq ($(DEVICEATLAS_SRC),)
+dadwsch:dadwsch.c
+ $(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS)
+
+LDFLAGS += -lda
+else
+DEVICEATLAS_INC = $(DEVICEATLAS_SRC)
+DEVICEATLAS_LIB = $(DEVICEATLAS_SRC)
+CFLAGS += -DDA_REGEX_HDR=\"dac_pcre2.c\" -DDA_REGEX_TAG=2
+CFLAGS += -DMOBI_CURL -DMOBI_CURLSSET -DMOBI_GZ -DMOBI_ZIP
+CFLAGS += -I$(DEVICEATLAS_INC) -I$(CURL_INC) -I$(PCRE2DIR)
+LDFLAGS += $(CURL_LDFLAGS) $(PCRE2_LDFLAGS) -lz -lzip -lpthread
+
+dadwsch:dadwsch.c $(DEVICEATLAS_SRC)/dac.c $(DEVICEATLAS_SRC)/dasch.c $(DEVICEATLAS_SRC)/dadwarc.c $(DEVICEATLAS_SRC)/dadwcom.c $(DEVICEATLAS_SRC)/dadwcurl.c $(DEVICEATLAS_SRC)/json.c $(DEVICEATLAS_SRC)/Os/daunix.c
+ $(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS)
+endif
+
+ifeq ($(OS), Linux)
+LDFLAGS += -lrt
+endif
+ifeq ($(OS), SunOS)
+LDFLAGS += -lrt
+endif
+
+clean:
+ rm -f *.o
+ rm -f $(DEVICEATLAS_LIB)*.o
+ rm -f dadwsch
+
diff --git a/addons/deviceatlas/dadwsch.c b/addons/deviceatlas/dadwsch.c
new file mode 100644
index 0..e35566a3f
--- /dev/null
+++ b/addons/deviceatlas/dadwsch.c
@@ -0,0 +1,195 @@
+#define _GNU_SOURCE
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#define ATLASTOKSZ PATH_MAX
+#define ATLASMAPNM "/hapdeviceatlas"
+
+const char *__pgname;
+
+static struct {
+ da_dwatlas_t o;
+ int ofd;
+ void* atlasmap;
+} global_deviceatlassch = {
+ .ofd = -1,
+ .atlasmap = NULL
+};
+
+
+void usage(void)
+{
+ fprintf(stderr, "%s -u download URL [-d hour (in H:M:S format) current hour by default] [-p path for the downloaded file, /tmp by default]\n", __pgname);
+ exit(EXIT_FAILURE);
+}
+
+static size_t jsonread(void *ctx, size_t count, char *buf)
+{
+ return fread(buf, 1, count, ctx);
+}
+
+static da_status_t jsonseek(void *ctx, off_t pos)
+{
+ return fseek(ctx, pos, SEEK_SET) != -1 ? DA_OK : DA_SYS;
+}
+
+static void dadwlog(dw_config_t cfg, const char* msg)
+{
+ time_t now = time(NULL);
+ char buf[26] = {0};
+ ctime_r(, buf);
+ buf[24] = 0;
+ fprintf(stderr, "%s: %s\n", buf, msg);
+}
+
+static dw_status_t dadwnot(void *a, dw_config_t *cfg)
+{
+ da_dwatlas_t *o = (da_dwatlas_t *)a;
+ if (!o)
+ return DW_ERR;
+ char *e;
+ char jsondbuf[26] = {0}, buf[26] = {0}, atlasp[ATLASTOKSZ] = {0};
+ time_t now = time(NULL);
+ time_t jsond;
+ int fd = -1;
+ (void)a;
+ jsond = da_getdatacreation(>atlas);
+ dwgetfinalp(o->dcfg.info, atlasp, sizeof(atlasp));
+ ctime_r(, jsondbuf);
+ ctime_r(, buf);
+ jsondbuf[24] = 0;
+ buf[24] = 0;
+
+ printf("%s: data file generated on `%s`\n", buf, jsondbuf);
+ int val = 1;
+ unsigned char *ptr = (unsigned char *)global_deviceatlassch.atlasmap;
+ memset(ptr, 0, sizeof(atlasp));
+ strcpy(ptr, atlasp);
+ return DW_OK;
+}
+
+static da_status_t dadwinit(void)
+{
+ if ((global_deviceatlassch.ofd = shm_open(ATLASMAPNM, O_RDWR | O_CREAT, 0660)) == -1) {
+ fprintf(stderr, "%s\n", strerror(errno));
+ return DA_SYS;
+ }
+
+ if (ftruncate(global_deviceatlassch.ofd, ATLASTOKSZ) == -1) {
+ close(global_deviceatlassch.ofd);
+ return DA_SYS;
+ }
+ lseek(global_deviceatlassch.ofd, 0, SEEK_SET);
+ global_deviceatlassch.atlasmap = mmap(0, ATLASTOKSZ, PROT_READ | PROT_WRITE, M
Re: [EXTERNAL] [PATCH] BUILD/MINOR fix solaris build with clang
Ah right then I use illumos based systems (which comes from opensolaris). solaris 11 is a bit unstable on a VM without fixes. On Mon, 17 Jan 2022 at 18:38, Willy TARREAU wrote: > > On Mon, Jan 17, 2022 at 05:38:46PM +, David CARLIER wrote: > > Mostly gcc 7.5 and clang 9 nowadays. > > Sorry I was not clear :-) I meant, do you build on an old Sun, in an > x86 VM, with a real Solaris or an OpenSolaris (or some variant) etc. > > I used to keep my old Sun working for a while because SPARC was extremely > picky on alignment and could often detect issues that are harder to spot > on other archs. I gave up when the CPU more or less died (the usual SPARC > thing, L2 cache corruption all the time). I wondered if switching to x86 > to keep a Solaris would bring me any benefit or not in terms of coverage. > Maybe it can if it still spots build issues (and we have evports there), > but I'm wondering about the least invasive setup we can think about in > this case. > > Thanks! > Willy
Re: [EXTERNAL] [PATCH] MEDIUM: pool monitor memory pressure on macOs
Hi,
I kind of knew it would trigger some controversy (especially using a
so specific API as you highlighted.
About your remarks on the CFLAGS, I had it automatically in my shell
environment without realising it (the "block" were triggered in my
tests) and forgot to put it in the Makefile
but would have needed a proper clang detection beforehand.
Your last remark is a valid point, my local tests were simple but what
you highlight are very likely to happen.
So yes let put it at rest indeed, I agree.
Regards.
On Mon, 17 Jan 2022 at 17:05, Willy TARREAU wrote:
>
> Hi David,
>
> On Sat, Jan 08, 2022 at 07:30:24PM +, David CARLIER wrote:
> > Hi
> >
> > Here a proposal to monitor memory pressure level on macOs to then
> > trigger pools trimming when it is critical.
> >
> > Cheers.
> >
> > thanks.
> >
> >
> > From 6b93fc00168a4e6ff80609ceb64582fea8d96ca0 Mon Sep 17 00:00:00 2001
> > From: David CARLIER
> > Date: Sat, 8 Jan 2022 19:25:18 +
> > Subject: [PATCH] MEDIUM: pool catching memory pressure level from the system
> > on macOs.
> >
> > proposal to provide an additional trigger to relief the pressure on the
> > pools on macOs, if HAProxy is under critical memory pressure via the
> > dispatch API.
>
> For this one I have a few concerns:
>
> - on the build side, it uses some LLVM/Clang extensions ("blocks" API),
> the test only uses defined(__BLOCKS__) which is possibly OK on modern
> systems but I'd rather make sure we limit this test to LLVM only so
> as to make sure that it's not inherited from something totally different
> by accident;
>
> - on the build side again, my readings about the blocks API (which I
> never heard about before) indicates that one has to pass -fblocks to
> benefit from it, otherwise it's not used. Hence my understanding is
> that this block remains excluded from the build.
>
> - on the maintenance side, I feel a bit concerned by the introduction
> of exotic language extensions. Having to go through such an unusual
> syntax just to call a function instead of passing a function pointer
> looks inefficient at best (as it emits a dummy function that calls
> the first one), and less maintainable, so as much as possible I'd
> rather avoid this and just use a standard callback.
>
> - on the efficiency side, I'm a bit embarrassed. What do we define as
> "critical" here ? How will users adjust the thresholds (if any) ?
> How do we know that the preset thresholds will not cause extra
> latencies by releasing memory too often for example ? Prior to 2.4
> depending on the build models, we used to call pool_gc() when facing
> an allocation error, before trying again. Nowadays we do something
> smarter, we monitor the average usage and spikes in each pool and
> automatically release the cold areas, meaning that overall we use
> less memory on varying loads, and are even less likely to salvage
> extra memory if/when reaching a level considered critical.
>
> - last, we've faced deadlocks in the past between some pool_alloc()
> and other blocks due to them being performed under thread isolation,
> and here it makes me think that we could reintroduce random indirect
> calls to thread_isolate() while in the process of allocating areas,
> thus reintroduce the risk of potential deadlocks (i.e. when another
> thread waits on thread_isolate and it cannot progress because it
> waits on a lock we still hold).
>
> So I fear that there are more trouble to expect mid-term than benefits
> to win. I don't know if you have metrics which show significant benefits
> in using that that outweigh the issues above (especially the last one is
> really not trivial to overcome), but for now I'd rather not include such
> a change.
>
> Thanks,
> Willy
Re: [EXTERNAL] [PATCH] BUILD/MINOR fix solaris build with clang
Mostly gcc 7.5 and clang 9 nowadays. Cheers. On Mon, 17 Jan 2022 at 16:43, Willy TARREAU wrote: > > On Thu, Jan 13, 2022 at 07:20:57PM +, David CARLIER wrote: > > Hi, > > > > here a little patch for solaris based systems. > > Thanks David, now applied. > > By the way, just out of curiosity, what do you use nowadays to build on > Solaris ? > > Willy
[PATCH] BUILD/MINOR fix solaris build with clang
Hi, here a little patch for solaris based systems. Cheers. From 9a9acb9b1c4f82b19bd9d9323b1334ea122c4dea Mon Sep 17 00:00:00 2001 From: David Carlier Date: Thu, 13 Jan 2022 19:16:48 + Subject: [PATCH] BUILD/MINOR: fix solaris build with clang. clang 9 sets the level to POSIX 2004. --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 0bf04f4ec..54f3ba4f5 100644 --- a/Makefile +++ b/Makefile @@ -378,7 +378,7 @@ ifeq ($(TARGET),solaris) set_target_defaults = $(call default_opts, \ USE_POLL USE_TPROXY USE_LIBCRYPT USE_CRYPT_H USE_GETADDRINFO USE_THREAD \ USE_RT USE_OBSOLETE_LINKER USE_EVPORTS USE_CLOSEFROM) - TARGET_CFLAGS = -DFD_SETSIZE=65536 -D_REENTRANT -D_XOPEN_SOURCE=500 -D__EXTENSIONS__ + TARGET_CFLAGS = -DFD_SETSIZE=65536 -D_REENTRANT -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ TARGET_LDFLAGS = -lnsl -lsocket endif -- 2.34.1
[PATCH] MEDIUM: pool monitor memory pressure on macOs
Hi Here a proposal to monitor memory pressure level on macOs to then trigger pools trimming when it is critical. Cheers. thanks. 0001-MEDIUM-pool-catching-memory-pressure-level-from-the-.patch Description: Binary data
Re: [EXTERNAL] Re: [PATCH] BUILD/MINOR: cpuset Fix FreeBSD 14 build
Ah right make sense indeed. On Sat, 8 Jan 2022 at 09:42, Илья Шипицин wrote: > > David, there are minor failures after this patch > BUG/MEDIUM: mworker: don't use _getsocks in wait mode · > haproxy/haproxy@f82afbb (github.com) > > пт, 31 дек. 2021 г. в 09:33, David CARLIER : >> >> Hi, >> >> all CPU macros which were incompatible with Linux (ie CPU_AND*, >> CPU_*OR...) had been changed and there is no backward compatibility >> with the old BSD api (at least not for now, they might introduce >> detection like solaris does for ex, FreeBSD 14 is not yet released) >> ultimately once FreeBSD 14 will become the LTS version the HAProxy >> cpuset code could be greatly simplified. >> I would suggest to backport up to the 2.4, that should be fine. >> >> Thanks. >> >> On Fri, 31 Dec 2021 at 06:18, Willy TARREAU wrote: >> > >> > Hi David, >> > >> > On Fri, Dec 31, 2021 at 05:02:37AM +, David CARLIER wrote: >> > > Here a simpler version if that s fine with you. >> > >> > Indeed, even simpler and I also prefer this one. I've now merged it, >> > thank you. Just to be sure, was this a breaking change in FreeBSD or >> > is the old API still supported ? I'm trying to figure if we have to >> > backport this one to stable releases in fact. >> > >> > BTW, I still hadn't had a look at your BPF patchset by lack of time, >> > but I will do soon. Rest assured they're not lost :-) >> > >> > Thanks, >> > Willy >> 0001-BUILD-MINOR-haproxy-build-fix-introduced-by-previous.patch Description: Binary data
[PATCH] MINOR haproxy switching to sched_setaffinity on FreeBSD 14
Hi here a little patch proposal. Kind regards. Thanks. 0001-MINOR-haproxy-switching-to-sched_setaffinity-for-Fre.patch Description: Binary data
[PATHC] BUILD/MINOR: solaris build fix
Hi here a minor patch for solaris based system. not urgent tough happy new year in advance :) From bc3e1174ca9a01e4a80972d28d452fff1907fc00 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Fri, 31 Dec 2021 08:15:29 + Subject: [PATCH] BUILD/MINOR: tools solaris build fix. dladdr takes a mutable address on this platform. --- src/tools.c | 4 1 file changed, 4 insertions(+) diff --git a/src/tools.c b/src/tools.c index c60f80b27..aa90341b2 100644 --- a/src/tools.c +++ b/src/tools.c @@ -4819,8 +4819,12 @@ static int dladdr_and_size(const void *addr, Dl_info *dli, size_t *size) ret = dladdr1(addr, dli, (void **), RTLD_DL_SYMENT); if (ret) *size = sym ? sym->st_size : 0; +#else +#if defined(__sun) + ret = dladdr((void *)addr, dli); #else ret = dladdr(addr, dli); +#endif *size = 0; #endif return ret; -- 2.34.1
Re: [EXTERNAL] Re: [PATCH] BUILD/MINOR: cpuset Fix FreeBSD 14 build
Hi, all CPU macros which were incompatible with Linux (ie CPU_AND*, CPU_*OR...) had been changed and there is no backward compatibility with the old BSD api (at least not for now, they might introduce detection like solaris does for ex, FreeBSD 14 is not yet released) ultimately once FreeBSD 14 will become the LTS version the HAProxy cpuset code could be greatly simplified. I would suggest to backport up to the 2.4, that should be fine. Thanks. On Fri, 31 Dec 2021 at 06:18, Willy TARREAU wrote: > > Hi David, > > On Fri, Dec 31, 2021 at 05:02:37AM +, David CARLIER wrote: > > Here a simpler version if that s fine with you. > > Indeed, even simpler and I also prefer this one. I've now merged it, > thank you. Just to be sure, was this a breaking change in FreeBSD or > is the old API still supported ? I'm trying to figure if we have to > backport this one to stable releases in fact. > > BTW, I still hadn't had a look at your BPF patchset by lack of time, > but I will do soon. Rest assured they're not lost :-) > > Thanks, > Willy
Re: [PATCH] BUILD/MINOR: cpuset Fix FreeBSD 14 build
Here a simpler version if that s fine with you. Cheers. On Thu, 30 Dec 2021 at 16:10, David CARLIER wrote: > > Hi > > Here a little build fix for FreeBSD 14. > > Cheers. > > Kind regards. 0001-BUILD-MINOR-cpuset-FreeBSD-14-build-fix.patch Description: Binary data
Re: [PATCH] BUILD/MINOR: cpuset Fix FreeBSD 14 build
I would suggest at least 12.3 at least if available on cirrus. On Thu, 30 Dec 2021 at 17:17, Илья Шипицин wrote: > > we use FreeBSD 12 for CI > haproxy/.cirrus.yml at master · haproxy/haproxy (github.com) > > David, can you have a look please ? Should we switch to newer version? > > чт, 30 дек. 2021 г. в 19:13, David CARLIER : >> >> Hi >> >> Here a little build fix for FreeBSD 14. >> >> Cheers. >> >> Kind regards.
[PATCH] BUILD/MINOR: cpuset Fix FreeBSD 14 build
Hi Here a little build fix for FreeBSD 14. Cheers. Kind regards. 0001-BUILD-MINOR-cpuset-Fix-FreeBSD-14-build.patch Description: Binary data
[PATCH 3/3] socket add support for socket BPF filter related to SO_REUSEPORT
From 44f69184b7c10373b9de53acba09b24ee83b2af8 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Fri, 17 Dec 2021 17:24:26 +
Subject: [PATCH 3/3] MEDIUM: socket Adding support for the new reuseportbpf
directive.
documentation part.
---
doc/configuration.txt | 5 +
1 file changed, 5 insertions(+)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index b739df7c0..e71e7fade 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -1077,6 +1077,7 @@ The following keywords are supported in the "global" section :
- nosplice
- nogetaddrinfo
- noreuseport
+ - reuseportbpf
- profiling.tasks
- spread-checks
- server-state-base
@@ -2421,6 +2422,10 @@ noreuseport
Disables the use of SO_REUSEPORT - see socket(7). It is equivalent to the
command line argument "-dR".
+reuseportbpf
+ Enables the use of SO_ATTACH_REUSEPORT_CBPF and if SO_REUSEPORT is enabled
+ on Linux (4.6 and above).
+
profiling.memory { on | off }
Enables ('on') or disables ('off') per-function memory profiling. This will
keep usage statistics of malloc/calloc/realloc/free calls anywhere in the
--
2.32.0
[PATCH 1/3] socket add support for socket BPF filter related to SO_REUSEPORT
Hi,
here a little patchset proposal to support cpu locality over a socket.
I would understand if it is rejected tough no problems.
Thanks.
From f207b3e1f4cfaf9a1682778be747378e7146ce82 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Fri, 17 Dec 2021 17:19:48 +
Subject: [PATCH 1/3] MINOR: socket Adds support got reuseportbpf new
configuration directive.
configuration parsing bit and new flag.
---
include/haproxy/compat.h | 4
include/haproxy/global-t.h | 3 +++
src/cfgparse-global.c | 5 +
3 files changed, 12 insertions(+)
diff --git a/include/haproxy/compat.h b/include/haproxy/compat.h
index f8afe1b5c..7d34ac4cb 100644
--- a/include/haproxy/compat.h
+++ b/include/haproxy/compat.h
@@ -159,6 +159,10 @@ typedef struct { } empty_t;
#include
#endif
+#if defined(SO_ATTACH_REUSEPORT_CBPF)
+#include
+#endif
+
/* On Linux, IP_TRANSPARENT and/or IP_FREEBIND generally require a kernel patch */
#if defined(USE_LINUX_TPROXY)
#if !defined(IP_FREEBIND)
diff --git a/include/haproxy/global-t.h b/include/haproxy/global-t.h
index fb01e0768..8d65df4ee 100644
--- a/include/haproxy/global-t.h
+++ b/include/haproxy/global-t.h
@@ -72,6 +72,9 @@
#define GTUNE_SCHED_LOW_LATENCY (1<<19)
#define GTUNE_IDLE_POOL_SHARED (1<<20)
#define GTUNE_DISABLE_H2_WEBSOCKET (1<<21)
+#define GTUNE_USE_REUSEPORTBPF (1<<22)
+
+#define GTUNE_LAST (1<<31)
/* SSL server verify mode */
enum {
diff --git a/src/cfgparse-global.c b/src/cfgparse-global.c
index a9e1b9480..9bad9b547 100644
--- a/src/cfgparse-global.c
+++ b/src/cfgparse-global.c
@@ -167,6 +167,11 @@ int cfg_parse_global(const char *file, int linenum, char **args, int kwm)
goto out;
global.tune.options &= ~GTUNE_USE_REUSEPORT;
}
+ else if (strcmp(args[0], "reuseportbpf") == 0) {
+ if (alertif_too_many_args(0, file, linenum, args, _code))
+ goto out;
+ global.tune.options |= GTUNE_USE_REUSEPORTBPF;
+ }
else if (strcmp(args[0], "quiet") == 0) {
if (alertif_too_many_args(0, file, linenum, args, _code))
goto out;
--
2.32.0
[PATCH 2/3] socket add support for socket BPF filter related to SO_REUSEPORT
From a7a48f5146d9a1784c8e1febc229f309814efa11 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Fri, 17 Dec 2021 17:21:10 +
Subject: [PATCH 2/3] MEDIUM: socket Add support for the new reuseportbpf
directive.
It's an option supported from Linux 4.6, we purposely deactivate by default
even if SO_REUSEPORT is on, it might be benefitial only in few chosen
kernel versions/hardware cases in the end.
Starting with a basic BPF socket filter, pulling the processor id explicitly,
it could be eventually expanded if needs arise.
---
src/sock_inet.c | 35 +--
1 file changed, 33 insertions(+), 2 deletions(-)
diff --git a/src/sock_inet.c b/src/sock_inet.c
index fa04dfed0..d8c50c0b5 100644
--- a/src/sock_inet.c
+++ b/src/sock_inet.c
@@ -335,8 +335,39 @@ int sock_inet_bind_receiver(struct receiver *rx, char **errmsg)
/* OpenBSD and Linux 3.9 support this. As it's present in old libc versions of
* Linux, it might return an error that we will silently ignore.
*/
- if (!ext && (global.tune.options & GTUNE_USE_REUSEPORT))
- setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, , sizeof(one));
+ if (!ext && (global.tune.options & GTUNE_USE_REUSEPORT)) {
+ int ret __maybe_unused;
+ ret = setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, , sizeof(one));
+#ifdef SO_ATTACH_REUSEPORT_CBPF
+ /* Linux 4.6 supports this to have a tigher control over the SO_REUSEPORT algorithm
+ * here an attempt to guarantee for the socket to be bound to a particular cpu
+ * rather than the actual round robin distribution.
+ *
+ * option disabled by default as this might not necessarily translate to nicer performances,
+ * would need broader tests on various configurations, while kernel support evolves, to push otherwise.
+ */
+ if (ret != -1 && (global.tune.options & GTUNE_USE_REUSEPORTBPF)) {
+ static struct sock_filter hacode[2] = {
+ /* processor cpu id into A */
+{ BPF_LD | BPF_W | BPF_ABS, 0, 0, (uint32_t)(SKF_AD_OFF + SKF_AD_CPU) },
+ /* It could have been broadened as below, but it is a configuration bit */
+ /* modulo threads */
+ /* { BPF_ALU | BPF_MOD, 0, 0, (uint32_t)() }, */
+ /* returns A */
+{ BPF_RET | BPF_A, 0, 0, 0 },
+ };
+ static struct sock_fprog haprog = {
+.filter = hacode,
+.len = (sizeof(hacode) / sizeof(hacode[0])),
+ };
+
+ if (setsockopt(fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, , sizeof(haprog)) == -1) {
+err |= ERR_FATAL | ERR_ALERT;
+memprintf(errmsg, "cannot do reuseport bpf");
+ }
+ }
+#endif
+ }
#endif
if (!ext && (rx->settings->options & RX_O_FOREIGN)) {
--
2.32.0
Re: [PATCH] MEDIUM numa supports for FreeBSD
Cheers ! On Wed, 15 Dec 2021 at 10:15, Amaury Denoyelle wrote: > > On Tue, Dec 14, 2021 at 02:12:28AM +, David CARLIER wrote: > > ping :) > > On Mon, 6 Dec 2021 at 11:07, David CARLIER wrote: > > > > > > Hi > > > > > > Here a little patch for proper NUMA topology detection on FreeBSD. > > > > > > Thanks. > > > > > > Regards. > > I have merged your patch, thanks for your contribution. Note that I have > splitted it in two to facilitate possible revert or cherry-picking of > numa_detect_topology() various implementation. I also used the BUG_ON > macro which is the idiomatic way for asserts in haproxy. > > Sorry for the delay, and thanks again, > > -- > Amaury Denoyelle
Re: [PATCH] MEDIUM numa supports for FreeBSD
ping :) On Mon, 6 Dec 2021 at 11:07, David CARLIER wrote: > > Hi > > Here a little patch for proper NUMA topology detection on FreeBSD. > > Thanks. > > Regards.
[PATCH] MEDIUM numa supports for FreeBSD
Hi
Here a little patch for proper NUMA topology detection on FreeBSD.
Thanks.
Regards.
From 1fc467bf273e66b82497f28816ee3b03c88067e2 Mon Sep 17 00:00:00 2001
From: David CARLIER
Date: Mon, 6 Dec 2021 11:00:10 +
Subject: [PATCH] MEDIUM: numa detect topology on FreeBSD.
allowing for all platforms supporting cpu affinity to have a chance
to detect the cpu topology from a given valid node (e.g.
DragonflyBSD seems to be NUMA aware from a kernel's perspective
and seems to be willing start to provide userland means to get
proper info).
---
include/haproxy/cpuset-t.h | 1 +
src/cfgparse.c | 54 --
2 files changed, 53 insertions(+), 2 deletions(-)
diff --git a/include/haproxy/cpuset-t.h b/include/haproxy/cpuset-t.h
index 541fb75cd..b26da7245 100644
--- a/include/haproxy/cpuset-t.h
+++ b/include/haproxy/cpuset-t.h
@@ -9,6 +9,7 @@
#ifdef __FreeBSD__
#include
#include
+#include
#include
#endif
#endif
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 06352e294..a23c2f713 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -33,6 +33,7 @@
#include
#include
#include
+#include
#include
#include
@@ -2212,7 +2213,8 @@ int readcfgfile(const char *file)
return err_code;
}
-#if defined(USE_THREAD) && defined(__linux__) && defined USE_CPU_AFFINITY
+#if defined(USE_THREAD) && defined USE_CPU_AFFINITY
+#if defined(__linux__)
/* filter directory name of the pattern node */
static int numa_filter(const struct dirent *dir)
{
@@ -2372,7 +2374,55 @@ static int numa_detect_topology()
return ha_cpuset_count(_cpu_set);
}
+#elif defined(__FreeBSD__)
+static int numa_detect_topology()
+{
+ struct hap_cpuset node_cpu_set;
+ size_t ndomains = 0, i;
+ size_t len = sizeof(ndomains);
+
+ if (sysctlbyname("vm.ndomains", , , NULL, 0) == -1) {
+ ha_notice("Cannot assess the number of CPUs domains\n");
+ return 0;
+ }
+
+ assert(ndomains <= MAXMEMDOM);
+ ha_cpuset_zero(_cpu_set);
+
+ /*
+ * We retrieve the first active valid CPU domain
+ * with active cpu and binding it, we returns
+ * the number of cpu from the said domain
+ */
+ for (i = 0; i < ndomains; i ++) {
+ struct hap_cpuset dom;
+ ha_cpuset_zero();
+ if (cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_DOMAIN, i, sizeof(dom.cpuset), ) == -1)
+ continue;
+
+ if (!ha_cpuset_count())
+ continue;
+
+ ha_cpuset_assign(_cpu_set, );
+
+ if (cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_PID, -1, sizeof(node_cpu_set.cpuset), _cpu_set.cpuset) == -1) {
+ ha_warning("Cannot set the cpu affinity for this multi-cpu machine\n");
+
+ /* clear the cpuset used as return value */
+ ha_cpuset_zero(_cpu_set);
+ }
+ break;
+ }
+
+ return ha_cpuset_count(_cpu_set);
+}
+#else
+static int numa_detect_topology()
+{
+ return 0;
+}
#endif /* __linux__ && USE_CPU_AFFINITY */
+#endif
/*
* Returns the error code, 0 if OK, or any combination of :
@@ -2425,7 +2475,7 @@ int check_config_validity()
#if defined(USE_THREAD)
{
int numa_cores = 0;
-#if defined(__linux__) && defined USE_CPU_AFFINITY
+#if defined(USE_CPU_AFFINITY)
if (global.numa_cpu_mapping && !thread_cpu_mask_forced())
numa_cores = numa_detect_topology();
#endif
--
2.34.1
Re: [PATCH] MEDIUM: pool second update but for the apple case
Sorry the same patch removing useless line :) On Fri, 26 Nov 2021 at 20:51, David CARLIER wrote: > > Hi > > Here the following up for the apple system allocator (aka libmalloc). > > Cheers. > > Kind regards. 0001-MEDIUM-pool-Following-up-on-previous-pool-trimming-u.patch Description: Binary data
[PATCH] MEDIUM: pool second update but for the apple case
Hi Here the following up for the apple system allocator (aka libmalloc). Cheers. Kind regards. 0001-MEDIUM-pool-Following-up-on-previous-pool-trimming-u.patch Description: Binary data
Re: [PATCH]: MEDIUM: pool little FreeBSD support improvement.
Ok I applied your suggestions and move back the malloc_trim/mallinfo
part as it was before.
On Thu, 25 Nov 2021 at 14:37, Willy Tarreau wrote:
>
> On Thu, Nov 25, 2021 at 01:19:39PM +, David CARLIER wrote:
> > Here a patchset instead :)
>
> Thanks!
>
> I've reviewed it, I'm having some comments below:
>
> > From e8daa477b53a43ab39113cf0e9c43d9bbda1e9a9 Mon Sep 17 00:00:00 2001
> > From: David Carlier
> > Date: Thu, 25 Nov 2021 10:26:50 +
> > Subject: [PATCH 1/3] MINOR: pool: refactoring to make it available for other
> > systems/allocators.
> >
> > Actually it's focusing on linux/glibc, we re trying to expand it for other
> > possible systems and allocators combinations.
> > ---
> > include/haproxy/compat.h | 9 +++--
> > src/pool.c | 38 +++---
> > 2 files changed, 14 insertions(+), 33 deletions(-)
> >
> > diff --git a/include/haproxy/compat.h b/include/haproxy/compat.h
> > index 25b15a1f0..b801dac08 100644
> > --- a/include/haproxy/compat.h
> > +++ b/include/haproxy/compat.h
> > @@ -263,9 +263,14 @@ typedef struct { } empty_t;
> > #endif
> >
> > /* glibc 2.33 provides mallinfo2() that overcomes mallinfo()'s type
> > limitations */
> > -#if (defined(__GNU_LIBRARY__) && (__GLIBC__ > 2 || __GLIBC__ == 2 &&
> > __GLIBC_MINOR__ >= 33))
> > +#if defined(__GNU_LIBRARY__)
> > #include
> > -#define HA_HAVE_MALLINFO2
> > +#if (__GLIBC__ > 2 || __GLIBC__ == 2 && __GLIBC_MINOR__ >= 33)
> > +#define HA_MALLINFO_API mallinfo2
> > +#else
> > +#define HA_MALLINFO_API mallinfo
> > +#endif
> > +#define HA_HAVE_MALLINFO
> > #endif
> >
> > /* FreeBSD also has malloc_usable_size() but it requires malloc_np.h */
> > diff --git a/src/pool.c b/src/pool.c
> > index af46b4469..16ef76cf0 100644
> > --- a/src/pool.c
> > +++ b/src/pool.c
> > @@ -42,14 +42,15 @@ int mem_poison_byte = -1;
> > static int mem_fail_rate = 0;
> > #endif
> >
> > -#if defined(HA_HAVE_MALLOC_TRIM)
> > static int using_libc_allocator = 0;
> >
> > /* ask the allocator to trim memory pools */
> > static void trim_all_pools(void)
> > {
> > +#ifdef HAVE_MALLOC_TRIM
> > if (using_libc_allocator)
> > malloc_trim(0);
> > +#endif
> > }
> >
> > /* check if we're using the same allocator as the one that provides
> > @@ -60,48 +61,23 @@ static void trim_all_pools(void)
> > */
> > static void detect_allocator(void)
> > {
> > -#ifdef HA_HAVE_MALLINFO2
> > - struct mallinfo2 mi1, mi2;
> > -#else
> > - struct mallinfo mi1, mi2;
> > -#endif
> > +#if defined(HA_HAVE_MALLINFO)
> > + struct HA_MALLINFO_API mi1, mi2;
> > void *ptr;
> >
> > -#ifdef HA_HAVE_MALLINFO2
> > - mi1 = mallinfo2();
> > -#else
> > - mi1 = mallinfo();
> > -#endif
> > + mi1 = HA_MALLINFO_API();
> > ptr = DISGUISE(malloc(1));
> > -#ifdef HA_HAVE_MALLINFO2
> > - mi2 = mallinfo2();
> > -#else
> > - mi2 = mallinfo();
> > -#endif
> > + mi2 = HA_MALLINFO_API();
> > free(DISGUISE(ptr));
> >
> > using_libc_allocator = !!memcmp(, , sizeof(mi1));
> > +#endif
> > }
>
> I find it unwelcome to use a single macro for the struct name and
> the function. It just turns out that they are the same but they're
> for different object types, and if one day we have mallinfo3() that
> works on struct mallinfo2, we're doomed.
I hope it will never come to such things, sounds terrible to me :)
even tough I have hard time to imagine a situation like this.
And the gain in legibility
> is not that great overall, I'd rather keep them with the ugly ifdefs
> here. If you prefer, just write two blocks in the function, one for
> mallinfo2 and one for the other. That's what I did the first time
> but wasn't much convinced about the benefit, but at this point it's
> a matter of taste.
>
> If you really want to have it this way, otherwise use two different
> macros, e.g.:
> #define HA_MALLINFO_TYPE struct mallinfo
> #define HA_MALLINFO_FUNC(x) mallinfo(x)
>
> But again this adds abstraction that doesn't help much, especially in
> tricky code which is written to diagnose certain things.
>
> > From c0f5cf6f935392e1ea69a84aae104de5ed06e68c Mon Sep 17 00:00:00 2001
> > From: David Carlier
> > Date: Thu, 25 Nov 2021 13:00:54 +
> > Subject: [PATCH 2/3] MEDIUM: pool : detecting jemalloc usage at run
Re: [PATCH]: MEDIUM: pool little FreeBSD support improvement.
On Thu, 25 Nov 2021 at 12:25, Willy Tarreau wrote:
>
> On Thu, Nov 25, 2021 at 04:38:27PM +0500, ??? wrote:
> > > Thus I think that instead of focusing on the OS we ought to continue
> > > to focus on the allocator and improve runtime detection:
> > >
> > > - glibc (currently detected using detect_allocator)
> > > => use malloc_trim()
> > > - jemalloc at build time (mallctl != NULL)
> > > => use mallctl() as you did
> > > - jemalloc at runtime (mallctl == NULL but dlsym("mallctl") != NULL)
> > > => use mallctl() as you did
> > > - others
> > > => no trimming
> > >
> >
> > I never imagined earlier that high level applications (such as reverse
> > https/tcp proxy) cares about such low level things as allocator behaviour.
> > no jokes, really.
>
> Yes it does count a lot. That's also why we spent a lot of time optimizing
> the pools, to limit the number of calls to the system's allocator for
> everything that uses a fixed size. I've seen some performance graphs in
> our internal ticket tracker showing the memory consumption between and
> after the switch to jemalloc, and the CPU usage as well, and sometimes
> it was very important.
>
> Glibc improved quite a bit recently (2.28 or 2.33 I don't remember) by
> implementing a per-thread cache in its ptmalloc. But in our case it's
> still not as good as jemalloc, and neither perform as well as our
> thread-local pools for fixed sizes.
>
> I'm seeing in a paper about snmalloc that it performs exceptionally well
> for small allocations. I just don't know how this degrades depending on
> the access patterns. For example some allocators are fast when you free()
> in the exact reverse allocation order, but can start to fragment or have
> more work to do finding holes if you don't free() in the exact same order.
>
If you re curious there is also mimalloc (with a pretty rich C api)
from Microsoft too.
> But that's something to keep an eye on in the future.
>
> Willy
Re: [PATCH]: MEDIUM: pool little FreeBSD support improvement.
Here a patchset instead :)
On Thu, 25 Nov 2021 at 12:25, Willy Tarreau wrote:
>
> On Thu, Nov 25, 2021 at 04:38:27PM +0500, ??? wrote:
> > > Thus I think that instead of focusing on the OS we ought to continue
> > > to focus on the allocator and improve runtime detection:
> > >
> > > - glibc (currently detected using detect_allocator)
> > > => use malloc_trim()
> > > - jemalloc at build time (mallctl != NULL)
> > > => use mallctl() as you did
> > > - jemalloc at runtime (mallctl == NULL but dlsym("mallctl") != NULL)
> > > => use mallctl() as you did
> > > - others
> > > => no trimming
> > >
> >
> > I never imagined earlier that high level applications (such as reverse
> > https/tcp proxy) cares about such low level things as allocator behaviour.
> > no jokes, really.
>
> Yes it does count a lot. That's also why we spent a lot of time optimizing
> the pools, to limit the number of calls to the system's allocator for
> everything that uses a fixed size. I've seen some performance graphs in
> our internal ticket tracker showing the memory consumption between and
> after the switch to jemalloc, and the CPU usage as well, and sometimes
> it was very important.
>
> Glibc improved quite a bit recently (2.28 or 2.33 I don't remember) by
> implementing a per-thread cache in its ptmalloc. But in our case it's
> still not as good as jemalloc, and neither perform as well as our
> thread-local pools for fixed sizes.
>
> I'm seeing in a paper about snmalloc that it performs exceptionally well
> for small allocations. I just don't know how this degrades depending on
> the access patterns. For example some allocators are fast when you free()
> in the exact reverse allocation order, but can start to fragment or have
> more work to do finding holes if you don't free() in the exact same order.
>
> But that's something to keep an eye on in the future.
>
> Willy
From 7206aff8ce6b6cb8c9b9b36d33b4f218f33cf0d1 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Thu, 25 Nov 2021 13:14:37 +
Subject: [PATCH 3/3] MEDIUM: pool: using jemalloc when relevant.
Finally, we trim the arenas with jemalloc if detected otherwise falling back
to glibc's malloc_trim.
---
src/pool.c | 18 --
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/src/pool.c b/src/pool.c
index 65c037911..3ec7a5386 100644
--- a/src/pool.c
+++ b/src/pool.c
@@ -48,10 +48,24 @@ static int(*my_mallctl)(const char *, void *, size_t *, void *, size_t) = NULL;
/* ask the allocator to trim memory pools */
static void trim_all_pools(void)
{
+ if (using_libc_allocator) {
+ if (my_mallctl) {
+ unsigned int i, narenas = 0;
+ size_t len = sizeof(narenas);
+
+ if (my_mallctl("arenas.narenas", , , NULL, 0) == 0) {
+for (i = 0; i < narenas; i ++) {
+ char mib[32] = {0};
+ snprintf(mib, sizeof(mib), "arena.%u.purge", i);
+ (void)my_mallctl(mib, NULL, NULL, NULL, 0);
+}
+ }
+ } else {
#ifdef HAVE_MALLOC_TRIM
- if (using_libc_allocator)
- malloc_trim(0);
+ malloc_trim(0);
#endif
+ }
+ }
}
/* check if we're using the same allocator as the one that provides
--
2.33.1
From e8daa477b53a43ab39113cf0e9c43d9bbda1e9a9 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Thu, 25 Nov 2021 10:26:50 +
Subject: [PATCH 1/3] MINOR: pool: refactoring to make it available for other
systems/allocators.
Actually it's focusing on linux/glibc, we re trying to expand it for other possible systems and allocators combinations.
---
include/haproxy/compat.h | 9 +++--
src/pool.c | 38 +++---
2 files changed, 14 insertions(+), 33 deletions(-)
diff --git a/include/haproxy/compat.h b/include/haproxy/compat.h
index 25b15a1f0..b801dac08 100644
--- a/include/haproxy/compat.h
+++ b/include/haproxy/compat.h
@@ -263,9 +263,14 @@ typedef struct { } empty_t;
#endif
/* glibc 2.33 provides mallinfo2() that overcomes mallinfo()'s type limitations */
-#if (defined(__GNU_LIBRARY__) && (__GLIBC__ > 2 || __GLIBC__ == 2 && __GLIBC_MINOR__ >= 33))
+#if defined(__GNU_LIBRARY__)
#include
-#define HA_HAVE_MALLINFO2
+#if (__GLIBC__ > 2 || __GLIBC__ == 2 && __GLIBC_MINOR__ >= 33)
+#define HA_MALLINFO_API mallinfo2
+#else
+#define HA_MALLINFO_API mallinfo
+#endif
+#define HA_HAVE_MALLINFO
#endif
/* FreeBSD also has malloc_usable_size() but it requires malloc_np.h */
diff --git a/src/pool.c b/src/pool.c
index af46b4469..16ef76cf0 100644
--- a/src/pool.c
+++ b/src/pool.c
@@ -42,14 +42,15 @@ int mem_poison_byte = -1;
static int mem_fail_rate = 0;
#endif
-#if defined(HA_HAVE_MALLOC_TRIM)
static int using_libc_allocator = 0;
/* ask the allocator to trim memory pools *
Re: [PATCH]: MEDIUM: pool little FreeBSD support improvement.
On Thu, 25 Nov 2021 at 11:38, Илья Шипицин wrote:
>
>
>
> чт, 25 нояб. 2021 г. в 14:54, Willy Tarreau :
>>
>> Hi David,
>>
>> On Wed, Nov 24, 2021 at 08:08:39PM +, David CARLIER wrote:
>> > Hi
>> >
>> > here a little patch for FreeBSD to support memory arenas trimming.
>> (...)
>> > FreeBSD uses a slighty simplified version of jemalloc as libc allocator
>> > since many years (there is thoughts to eventually switch to snmalloc
>> > but not before a long time).
>> > We detect the libc in the least hacky way in this case aiming as jemalloc
>> > specific API then we try to purge arenas as much as we can.
>>
>> This one is interesting because according to the jemalloc doc on
>> my machine it could also work on linux with jemalloc, provided that
>> jemalloc is linked at build time (otherwise mallctl remains NULL).
>> However it remains available uding dlopen().
>>
>> Thus I think that instead of focusing on the OS we ought to continue
>> to focus on the allocator and improve runtime detection:
>>
>> - glibc (currently detected using detect_allocator)
>> => use malloc_trim()
>> - jemalloc at build time (mallctl != NULL)
>> => use mallctl() as you did
>> - jemalloc at runtime (mallctl == NULL but dlsym("mallctl") != NULL)
>> => use mallctl() as you did
>> - others
>> => no trimming
>
Fair enough I thought it was solely about system libc, but indeed the
Linux/jemalloc
combination is very common, interesting to look at indeed.
Ok will provide the patchsets later on :-)
>
> I never imagined earlier that high level applications (such as reverse
> https/tcp proxy) cares about such low level things as allocator behaviour.
> no jokes, really.
>
This is how you stand out ;-)
>
>>
>>
>> That would cover the vast majority of use cases where trimming is relevant.
>> I'm quite interested in jemalloc on linux, including when used at runtime
>> using LD_PRELOAD, because I see it quite frequently in high-performance
>> environments. When site operators see their memory usage fall two-fold and
>> the CPU as well by just prepending an LD_PRELOAD in their startup script,
>> they don't need to think twice and it's quickly done.
>>
>> Thus I think we could proceed like this in the pools init code:
>>
>>void detect_allocator()
>>{
>> extern int mallctl(const char *, void *, size_t *, void *, size_t)
>> __attribute__((weak));
>>
>> my_mallctl = mallctl;
>> if (!my_mallctl)
>> my_mallctl = get_sym_curr_addr("mallctl");
>>
>> if (!my_mallctl) {
>> /* existing tests based on mallinfo/mallinfo2 */
>> }
>>}
>>
>> At this point we'd have:
>> - my_mallctl not null when using jemalloc, hence trim() would always
>> use your code that purges all arenas
>>
>> - otherwise if using_libc_allocator is set, we can call malloc_trim()
>>
>> - otherwise it's an unknown allocator and we don't do anything.
>>
>> Thus could you please turn your patch into a small series that does
>> this ? The steps I'm seeing are:
>> 1. changing the detection logic so that it's always performed, and
>> not just on HA_HAVE_MALLOC_TRIM. This means the variants of
>> functions is_trim_enabled, trim_all_pools and detect_allocator()
>> are all remerged into a single variant. trim_all_pools() will be
>> the only one to rely on HA_HAVE_MALLOC_TRIM.
>>
>> 2. introduce mallctl availability detection and put its pointer into
>> a separate one that we can get unsing dlfcn when linked in.
>>
>> 3. modify trim_all_pools() to use mallctl() when available, otherwise
>> rely on current code with malloc_trim().
>>
>> This should make a significant improvement with plug-n-play detection
>> for the majority of users.
>>
>> Then if you want to introduce runtime detection for snmalloc to do the
>> same, this would make everything much easier.
>>
>> Thanks,
>> Willy
>>
[PATCH]: MEDIUM: pool little FreeBSD support improvement.
Hi
here a little patch for FreeBSD to support memory arenas trimming.
Thanks.
regards.
From 1d6386a626f56ca64c25e2dfbf2f9d90a81bd7ae Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Wed, 24 Nov 2021 20:02:41 +
Subject: [PATCH] MEDIUM: pool: trimming arenas on FreeBSD.
FreeBSD uses a slighty simplified version of jemalloc as libc allocator
since many years (there is thoughts to eventually switch to snmalloc
but not before a long time).
We detect the libc in the least hacky way in this case aiming as jemalloc
specific API then we try to purge arenas as much as we can.
---
include/haproxy/compat.h | 2 +-
src/pool.c | 32
2 files changed, 25 insertions(+), 9 deletions(-)
diff --git a/include/haproxy/compat.h b/include/haproxy/compat.h
index 25b15a1f0..daa58be5d 100644
--- a/include/haproxy/compat.h
+++ b/include/haproxy/compat.h
@@ -269,7 +269,7 @@ typedef struct { } empty_t;
#endif
/* FreeBSD also has malloc_usable_size() but it requires malloc_np.h */
-#if defined(USE_MEMORY_PROFILING) && defined(__FreeBSD__) && (__FreeBSD_version >= 72)
+#if defined(__FreeBSD__) && (__FreeBSD_version >= 72)
#include
#endif
diff --git a/src/pool.c b/src/pool.c
index af46b4469..f3ea8c7a7 100644
--- a/src/pool.c
+++ b/src/pool.c
@@ -42,8 +42,8 @@ int mem_poison_byte = -1;
static int mem_fail_rate = 0;
#endif
-#if defined(HA_HAVE_MALLOC_TRIM)
static int using_libc_allocator = 0;
+#if defined(HA_HAVE_MALLOC_TRIM)
/* ask the allocator to trim memory pools */
static void trim_all_pools(void)
@@ -82,26 +82,42 @@ static void detect_allocator(void)
using_libc_allocator = !!memcmp(, , sizeof(mi1));
}
-
-static int is_trim_enabled(void)
-{
- return using_libc_allocator;
-}
#else
+#if defined(__FreeBSD__)
+extern void sdallocx(void *, size_t, int) __attribute__((weak));
+#endif
+
static void trim_all_pools(void)
{
+#if defined(__FreeBSD__)
+ if (using_libc_allocator) {
+ unsigned int narenas = 0;
+ size_t len = sizeof(narenas);
+
+ if (mallctl("arenas.narenas", , , NULL, 0) == 0) {
+ for (unsigned int i = 0; i < narenas; i ++) {
+char mib[32] = {0};
+snprintf(mib, sizeof(mib), "arena.%u.purge", i);
+(void)mallctl(mib, NULL, NULL, NULL, 0);
+ }
+ }
+ }
+#endif
}
static void detect_allocator(void)
{
+#if defined(__FreeBSD__)
+ using_libc_allocator = (sdallocx != NULL);
+#endif
}
+#endif
static int is_trim_enabled(void)
{
- return 0;
+ return using_libc_allocator;
}
-#endif
/* Try to find an existing shared pool with the same characteristics and
* returns it, otherwise creates this one. NULL is returned if no memory
--
2.33.1
Rectify your technical issues.
Hi there, After spending some time on Haproxy.com and have an idea that you could implement quickly that would help increase your natural traffic, ranking, social media & conversion rates. Feel free to say no, but allow me to ask, is increasing the above something you’re interested in? David Divone. 301-563-8574
[PATCH] BUILD/MINOR: cpuset Freebsd build fix
Hi here a patch to address freebsd build failure. Kind regards. From 97df7fccc4f6a1b5d2b2ae8050ce58f7666f6c27 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Sat, 30 Oct 2021 12:22:21 +0100 Subject: [PATCH] BUILD/MINOR: cpuset freebsd build fix. adding missing strings.h header. --- include/haproxy/cpuset-t.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/haproxy/cpuset-t.h b/include/haproxy/cpuset-t.h index 0d5b6c804..541fb75cd 100644 --- a/include/haproxy/cpuset-t.h +++ b/include/haproxy/cpuset-t.h @@ -9,6 +9,7 @@ #ifdef __FreeBSD__ #include #include +#include #endif #endif -- 2.33.0
Re: [PATCH] BUILD/MINOR: atomics: mac arm64 build fix
Hi Willy,
Ok with your changes suggestions even tough it seemed to work fine
with the raspberry/clang combination (failure might be due to apple
oddities here) but
anyway let s fall back to the builtin regardless.
Btw __atomic_compare_exchange wants a mutable parameter as third
argument thus the cast to avoid the spurious build warnings.
Thanks.
On Tue, 26 Oct 2021 at 09:41, Willy Tarreau wrote:
>
> Hi David,
>
> On Sat, Oct 23, 2021 at 05:12:18PM +0100, David CARLIER wrote:
> > > > diff --git a/include/haproxy/atomic.h b/include/haproxy/atomic.h
> > > > index 3198b381a..29a06c57b 100644
> > > > --- a/include/haproxy/atomic.h
> > > > +++ b/include/haproxy/atomic.h
> > > > @@ -698,7 +698,7 @@ __ha_barrier_atomic_full(void)
> > > > */
> > > > #define __ha_cpu_relax() ({ asm volatile("isb" ::: "memory"); 1; })
> > > >
> > > > -#if defined(__ARM_FEATURE_ATOMICS) // ARMv8.1-A atomics
> > > > +#if defined(__ARM_FEATURE_ATOMICS) && !defined(__APPLE__) // ARMv8.1-A
> > > > atomics
> > >
> > > Hmmm that's not expected at all, what error are you getting ? We're
> > > not doing anything special, so if the __ARM_FEATURE_ATOMICS macro
> > > is defined, we ought to have them.
> >
> > Note this works ok with gcc but not clang. But clang is the main
> > compiler, gcc is secondary on this platform (less well supported).
> >
> > I tried to convert to llvm asm-ism but seems applying direct
> > contiguous data values (ie cmp) to registers seems unsupported and
> > requires more instructions/operands thus
> > it is safer to just fallback to the builtin which just does that
> > anyway as you can see :
> >
> > `
> > 0x100087a3c <+28>: movx3, x5
> > 0x100087a40 <+32>: casp x2, x3, x6, x7, [x8]
> > 0x100087a44 <+36>: eorx8, x3, x5
> > 0x100087a48 <+40>: eorx9, x2, x4
> > 0x100087a4c <+44>: orrx8, x9, x8
> > 0x100087a50 <+48>: cbzx8, 0x100087a58 ; <+56> at
> > atomic.h:757:2
> > `
>
> I'm not opposed to falling back to the builtin, which is easier to
> maintain, my concern remains the condition to enable this, as
> "defined(APPLE)" is only about a vendor an not at all about a toolchain.
>
> If you identified that it fails to emit the assembly, it will be the
> same for other vendors and the common denominator will be clang/llvm.
> And given that it provides a working builtin, then I'd rather rely on
> clang detection to use the builtin.
>
> > > > -#elif defined(__SIZEOF_INT128__) &&
> > > > defined(__GCC_HAVE_SYNC_COMPARE_AND_SWAP_16) // no ARMv8.1-A atomics
> > > > but 128-bit atomics
> > > > +#elif defined(__SIZEOF_INT128__) &&
> > > > (defined(__GCC_HAVE_SYNC_COMPARE_AND_SWAP_16) || defined(__APPLE__)) //
> > > > no ARMv8.1-A atomics but 128-bit atomics
> > >
> > > I feel uncomfortable with adding this "||" in the expression, as we ought
> > > to condition this instructions to this. Are you sure this was needed ? Did
> > > you check the resulting code to make sure a CASP instruction was emitted ?
> > >
> > > Something that would be nice is to dump all defines:
> > >
> > > gcc -dM -E - < /dev/null
> > >
> > > (or use clang instead of gcc).
> >
> > mac m1 supports rightfully the 128 bits type.
> >
> > `
> > #define __SIZEOF_INT128__ 16
> > `
>
> No doubt about this, I was mostly referring to all the other ones. In
> fact, given that there is the CASP instruction for this, I'd rather
> switch to "|| defined(__ARM_FEATURE_ATOMICS)".
>
> > I just recall on arm64 __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16 is not even
> > present and indeed when I look up on both on my raspberry and my mac
> > m1 that is confirmed.
>
> I think the *_SYNC_* only applies to the old sync_* atomic builtins, and
> are probably irrelevant to the new ones, thus we should probably simplify
> it this way:
>
>#elif defined(__SIZEOF_INT128__) && defined(__ARM_FEATURE_ATOMICS)
>
> Willy
0001-BUILD-MINOR-atomics-mac-arm64-build-fix.patch
Description: Binary data
Re: [PATCH] BUILD/MINOR: atomics: mac arm64 build fix
On Sat, 23 Oct 2021 at 16:50, Willy Tarreau wrote:
>
> Hi David,
>
> On Sat, Oct 23, 2021 at 02:51:59PM +0100, David CARLIER wrote:
> > Hi,
> > Hopefully not too late for the 2.5 release :-)
>
> No worries, and fixes can be merged later anyway. I have some questions
> below.
>
> > From b9c083252bdabf2d0bbfffa1383453cdfd94ab13 Mon Sep 17 00:00:00 2001
> > From: David CARLIER
> > Date: Sat, 23 Oct 2021 14:43:42 +0100
> > Subject: [PATCH] BUILD/MINOR: atomics mac arm64 build fix
> >
> > the inline assembly is invalid for this platform thus falling back
> > to the builtin instead.
> >
> > ---
> > include/haproxy/atomic.h | 6 +++---
> > 1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/include/haproxy/atomic.h b/include/haproxy/atomic.h
> > index 3198b381a..29a06c57b 100644
> > --- a/include/haproxy/atomic.h
> > +++ b/include/haproxy/atomic.h
> > @@ -698,7 +698,7 @@ __ha_barrier_atomic_full(void)
> > */
> > #define __ha_cpu_relax() ({ asm volatile("isb" ::: "memory"); 1; })
> >
> > -#if defined(__ARM_FEATURE_ATOMICS) // ARMv8.1-A atomics
> > +#if defined(__ARM_FEATURE_ATOMICS) && !defined(__APPLE__) // ARMv8.1-A
> > atomics
>
> Hmmm that's not expected at all, what error are you getting ? We're
> not doing anything special, so if the __ARM_FEATURE_ATOMICS macro
> is defined, we ought to have them.
Note this works ok with gcc but not clang. But clang is the main
compiler, gcc is secondary on this platform (less well supported).
I tried to convert to llvm asm-ism but seems applying direct
contiguous data values (ie cmp) to registers seems unsupported and
requires more instructions/operands thus
it is safer to just fallback to the builtin which just does that
anyway as you can see :
`
0x100087a3c <+28>: movx3, x5
0x100087a40 <+32>: casp x2, x3, x6, x7, [x8]
0x100087a44 <+36>: eorx8, x3, x5
0x100087a48 <+40>: eorx9, x2, x4
0x100087a4c <+44>: orrx8, x9, x8
0x100087a50 <+48>: cbzx8, 0x100087a58 ; <+56> at
atomic.h:757:2
`
>
> > /* returns 0 on failure, non-zero on success */
> > static forceinline int __ha_cas_dw(void *target, void *compare, const void
> > *set)
> > @@ -738,7 +738,7 @@ static forceinline int __ha_cas_dw(void *target, void
> > *compare, const void *set)
> > return ret;
> > }
> >
> > -#elif defined(__SIZEOF_INT128__) &&
> > defined(__GCC_HAVE_SYNC_COMPARE_AND_SWAP_16) // no ARMv8.1-A atomics but
> > 128-bit atomics
> > +#elif defined(__SIZEOF_INT128__) &&
> > (defined(__GCC_HAVE_SYNC_COMPARE_AND_SWAP_16) || defined(__APPLE__)) // no
> > ARMv8.1-A atomics but 128-bit atomics
>
> I feel uncomfortable with adding this "||" in the expression, as we ought
> to condition this instructions to this. Are you sure this was needed ? Did
> you check the resulting code to make sure a CASP instruction was emitted ?
>
> Something that would be nice is to dump all defines:
>
> gcc -dM -E - < /dev/null
>
> (or use clang instead of gcc).
mac m1 supports rightfully the 128 bits type.
`
#define __SIZEOF_INT128__ 16
`
I just recall on arm64 __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16 is not even
present and indeed when I look up on both on my raspberry and my mac
m1 that is confirmed.
>
> > /* According to
> > https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
> > * we can use atomics on __int128. The availability of CAS is defined
> > there:
> > @@ -752,7 +752,7 @@ static forceinline int __ha_cas_dw(void *target, void
> > *compare, const void *set)
> > /* returns 0 on failure, non-zero on success */
> > static __inline int __ha_cas_dw(void *target, void *compare, const void
> > *set)
> > {
> > - return __atomic_compare_exchange((__int128*)target,
> > (__int128*)compare, (const __int128*)set,
> > + return __atomic_compare_exchange((__int128*)target,
> > (__int128*)compare, (__int128*)set,
>
> It's not correct to remove the const there, first because this value is
> not expected to be changed by the emitted instructions, and second because
> we're violating the function's contract which promises not to modify this
> argument. Are you sure it's not just a leftover of your debugging session ?
>
> Thanks,
> Willy
[PATCH] BUILD/MINOR: atomics: mac arm64 build fix
Hi, Hopefully not too late for the 2.5 release :-) Kind regards. 0001-BUILD-MINOR-atomics-mac-arm64-build-fix.patch Description: Binary data
[PATCH] BUILD/MINOR: healthcheck netbsd build fix
Hi here a little patch for netbsd system. Kind regards. From 9a6a01ed06bbacffe76c8e504cb5c6c243912b2d Mon Sep 17 00:00:00 2001 From: David Carlier Date: Wed, 20 Oct 2021 19:00:02 +0100 Subject: [PATCH] BUILD/MINOR: healthcheck: netbsd build fix. adding missing header for time based structs. --- include/haproxy/connection-t.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/haproxy/connection-t.h b/include/haproxy/connection-t.h index 0ba8555ff..25fc57b2c 100644 --- a/include/haproxy/connection-t.h +++ b/include/haproxy/connection-t.h @@ -24,6 +24,7 @@ #include #include +#include #include #include #include -- 2.33.0
[PATCH]: dragonflybsd build fix
Hi here a little fix proposal for this platform.
Cheers.
From 6cfa1fce839504e04584d1bfedee188bc21c32b1 Mon Sep 17 00:00:00 2001
From: DC
Date: Sat, 4 Sep 2021 09:58:57 +0100
Subject: [PATCH] BUILD/MINOR: dragonfly build fix.
build failure since __read_mostly is undefined.
---
include/haproxy/compiler.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/include/haproxy/compiler.h b/include/haproxy/compiler.h
index ca3b844c1..435f7a28f 100644
--- a/include/haproxy/compiler.h
+++ b/include/haproxy/compiler.h
@@ -91,6 +91,8 @@
/* use this attribute on a variable to move it to the read_mostly section */
#if !defined(__DragonFly__)
#define __read_mostly HA_SECTION("read_mostly")
+#else
+#define __read_mostly
#endif
/* This allows gcc to know that some locations are never reached, for example
--
2.31.1
Re: [PATCH]: MINOR: proc: making the process able to produce ore dump on FreeBSD
On Sat, 21 Aug 2021 at 21:08, David CARLIER wrote: > > ah yes true.. > > On Sat, 21 Aug 2021 at 21:06, Tim Düsterhus wrote: > > > > David, > > > > On 8/21/21 10:35 AM, David CARLIER wrote: > > > Here a little patch for FreeBSD systems. > > > > > > > > > > diff --git a/Makefile b/Makefile > > > index 84b1bc0ea..f6d813c27 100644 > > > --- a/Makefile > > > +++ b/Makefile > > > @@ -36,6 +36,7 @@ > > > # USE_ACCEPT4 : enable use of accept4() on linux. Automatic. > > > # USE_CLOSEFROM: enable use of closefrom() on *bsd, solaris. > > > Automatic. > > > # USE_PRCTL: enable use of prctl(). Automatic. > > > +# USE_PROCCTL : enable use of procctl(). > > > > Should it read "Automatic." at the end, because it appears to be > > automatically enabled for the freebsd target? > > > > Best regards > > Tim Düsterhus 0001-MINOR-proc-setting-the-process-to-produce-a-core-dum.patch Description: Binary data
Re: [PATCH]: MINOR: proc: making the process able to produce ore dump on FreeBSD
ah yes true.. On Sat, 21 Aug 2021 at 21:06, Tim Düsterhus wrote: > > David, > > On 8/21/21 10:35 AM, David CARLIER wrote: > > Here a little patch for FreeBSD systems. > > > > > > diff --git a/Makefile b/Makefile > > index 84b1bc0ea..f6d813c27 100644 > > --- a/Makefile > > +++ b/Makefile > > @@ -36,6 +36,7 @@ > > # USE_ACCEPT4 : enable use of accept4() on linux. Automatic. > > # USE_CLOSEFROM: enable use of closefrom() on *bsd, solaris. > > Automatic. > > # USE_PRCTL: enable use of prctl(). Automatic. > > +# USE_PROCCTL : enable use of procctl(). > > Should it read "Automatic." at the end, because it appears to be > automatically enabled for the freebsd target? > > Best regards > Tim Düsterhus
[PATCH]: MINOR: proc: making the process able to produce ore dump on FreeBSD
Hi
Here a little patch for FreeBSD systems.
Thanks.
Regards.
From 82d3338206dd25cb046032d1b8f005228b8d0a76 Mon Sep 17 00:00:00 2001
From: DC
Date: Sat, 21 Aug 2021 09:13:10 +0100
Subject: [PATCH] MINOR: proc: setting the process to produce a core dump on
FreeBSD.
using the procctl api to set the current process as traceable, thus being able to produce a core dump as well.
making it as compile option if not wished or using freebsd prior to 11.x (last no EOL release).
---
Makefile | 7 ---
src/haproxy.c | 9 +
2 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile
index 84b1bc0ea..f6d813c27 100644
--- a/Makefile
+++ b/Makefile
@@ -36,6 +36,7 @@
# USE_ACCEPT4 : enable use of accept4() on linux. Automatic.
# USE_CLOSEFROM: enable use of closefrom() on *bsd, solaris. Automatic.
# USE_PRCTL: enable use of prctl(). Automatic.
+# USE_PROCCTL : enable use of procctl().
# USE_ZLIB : enable zlib library support and disable SLZ
# USE_SLZ : enable slz library instead of zlib (default=enabled)
# USE_CPU_AFFINITY : enable pinning processes to CPU on Linux. Automatic.
@@ -311,8 +312,8 @@ use_opts = USE_EPOLL USE_KQUEUE USE_NETFILTER \
USE_GETADDRINFO USE_OPENSSL USE_LUA USE_ACCEPT4\
USE_CLOSEFROM USE_ZLIB USE_SLZ USE_CPU_AFFINITY USE_TFO USE_NS \
USE_DL USE_RT USE_DEVICEATLAS USE_51DEGREES USE_WURFL USE_SYSTEMD \
- USE_OBSOLETE_LINKER USE_PRCTL USE_THREAD_DUMP USE_EVPORTS USE_OT \
- USE_QUIC USE_PROMEX USE_MEMORY_PROFILING
+ USE_OBSOLETE_LINKER USE_PRCTL USE_PROCCTL USE_THREAD_DUMP \
+ USE_EVPORTS USE_OT USE_QUIC USE_PROMEX USE_MEMORY_PROFILING
Target system options
# Depending on the target platform, some options are set, as well as some
@@ -391,7 +392,7 @@ endif
ifeq ($(TARGET),freebsd)
set_target_defaults = $(call default_opts, \
USE_POLL USE_TPROXY USE_LIBCRYPT USE_THREAD USE_CPU_AFFINITY USE_KQUEUE \
-USE_ACCEPT4 USE_CLOSEFROM USE_GETADDRINFO)
+USE_ACCEPT4 USE_CLOSEFROM USE_GETADDRINFO USE_PROCCTL)
endif
# DragonFlyBSD 4.3 and above
diff --git a/src/haproxy.c b/src/haproxy.c
index 047861ca9..db957256e 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -70,6 +70,10 @@
#include
#endif
+#if defined(USE_PROCCTL)
+#include
+#endif
+
#ifdef DEBUG_FULL
#include
#endif
@@ -3401,6 +3405,11 @@ int main(int argc, char **argv)
if (prctl(PR_SET_DUMPABLE, 1, 0, 0, 0) == -1)
ha_warning("[%s.main()] Failed to set the dumpable flag, "
"no core will be dumped.\n", argv[0]);
+#elif defined(USE_PROCCTL)
+ int traceable = PROC_TRACE_CTL_ENABLE;
+ if (procctl(P_PID, getpid(), PROC_TRACE_CTL, ) == -1)
+ ha_warning("[%s.main()] Failed to set the traceable flag, "
+ "no core will be dumped.\n", argv[0]);
#endif
}
--
2.32.0
BUILD: tools: get the absolute path on FreeBSD
Hi,
same as earlier but for FreeBSD this time.
Thanks.
From 07be546638949864ef678b0e39871b7febceeaed Mon Sep 17 00:00:00 2001
From: DC
Date: Tue, 17 Aug 2021 12:55:49 +0100
Subject: [PATCH] BUILD: tools: get the absolute path of the current binary on
FreeBSD. FreeBSD stores the absolute path into the auxiliary vector as well.
---
src/tools.c | 14 ++
1 file changed, 14 insertions(+)
diff --git a/src/tools.c b/src/tools.c
index 545fd9e8d..1961689dd 100644
--- a/src/tools.c
+++ b/src/tools.c
@@ -16,6 +16,12 @@
#include
#endif
+#if defined(__FreeBSD__)
+#include
+#include
+extern void *__elf_aux_vector;
+#endif
+
#if defined(__NetBSD__)
#include
#include
@@ -4766,6 +4772,14 @@ const char *get_exec_path()
if (execfn && execfn != ENOENT)
ret = (const char *)execfn;
+#elif defined(__FreeBSD__)
+ Elf_Auxinfo *auxv;
+ for (auxv = __elf_aux_vector; auxv->a_type != AT_NULL; ++auxv) {
+ if (auxv->a_type == AT_EXECPATH) {
+ ret = (const char *)auxv->a_un.a_ptr;
+ break;
+ }
+ }
#elif defined(__NetBSD__)
AuxInfo *auxv;
for (auxv = _dlauxinfo(); auxv->a_type != AT_NULL; ++auxv) {
--
2.32.0
BUILD: tools: gets the absolute path on NetBSD
Hi,
here a little patch proposal.
Thanks in advance.
From 30a530f3b352e907a3dd9daebfda2bb94dccbdbc Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Tue, 17 Aug 2021 08:44:25 +0100
Subject: [PATCH] BUILD: tools: get the absolute path of the current binary on
NetBSD.
NetBSD stores the absolute path into the auxiliary vector as well.
---
src/tools.c | 13 +
1 file changed, 13 insertions(+)
diff --git a/src/tools.c b/src/tools.c
index bd6bf4edc..545fd9e8d 100644
--- a/src/tools.c
+++ b/src/tools.c
@@ -16,6 +16,11 @@
#include
#endif
+#if defined(__NetBSD__)
+#include
+#include
+#endif
+
#include
#include
#include
@@ -4761,6 +4766,14 @@ const char *get_exec_path()
if (execfn && execfn != ENOENT)
ret = (const char *)execfn;
+#elif defined(__NetBSD__)
+ AuxInfo *auxv;
+ for (auxv = _dlauxinfo(); auxv->a_type != AT_NULL; ++auxv) {
+ if (auxv->a_type == AT_SUN_EXECNAME) {
+ ret = (const char *)auxv->a_v;
+ break;
+ }
+ }
#endif
return ret;
}
--
2.32.0
Re: [PATCH] memprof fix OpenBSD build.
Ok fair point(s). On Mon, 26 Jul 2021 at 12:04, Willy Tarreau wrote: > > Hi David, > > On Sun, Jul 25, 2021 at 11:07:00AM +0100, David CARLIER wrote: > > +/* OpenBSD does not have anything close to malloc_usable_size, thus > > profiling will be wrong regardless */ > > +#if defined(USE_MEMORY_PROFILING) && defined(__OpenBSD__) > > +#undef USE_MEMORY_PROFILING > > +#endif > > I disagree with this one. It means that someone who forces > USE_MEMORY_PROFILING on this OS will get no error and will not know > that it was silently disabled. We precisely use build-time options to > make sure that the user has control over what is enabled or disabled. > This would be an exception in the middle of all other options, let's > not do it. > > If OpenBSD doesn't have malloc_usable_size(), there are two > solutions. Either the option must not be used there (which is the > default), or we decide that we know for sure that it's stored at > *((char *)ptr - sizeof(void *)) with a mask, and we dereference it > there, and then we can enable it. But while I started this way at > the beginning, I'd rather avoid doing this because that could > break for those linking with different allocators. > > Regards, > Willy
