add me to mailing list

[Randy Schneiderman]

Thanks,
___
Randy Schneiderman | Sr. Manager Service Design | IPsoft, Inc.
O: 212-708-5573 | C: 347-267-3915 | F: 212-708-5010 | 
Randy.Schneiderman@ipsoft
Upcoming PTO: 5/28 - 6/6

Re: [PATCH] Make sure all the pollers get fd updates

[Willy Tarreau]

On Thu, May 17, 2018 at 07:37:10PM +0200, Olivier Houchard wrote:
> Here is a patch that should do the same for 1.8.

Pretty cool, thank you. Now I think we'll have about everything we wanted
to emit a new 1.8.

Willy

[PATCH] BUG/MEDIUM: stick-tables: Decrement ref_cnt in table_* converters

[Daniel Corbett]

Hello,

When using table_* converters ref_cnt was incremented
and never decremented causing entries to not expire.

The root cause appears to be that stktable_lookup_key()
was called within all sample_conv_table_* functions which was
incrementing ref_cnt and not decrementing after completion.

Added stktable_release() to the end of each sample_conv_table_*
function.

This should be backported to 1.8.


Thanks,
-- Daniel

>From 28530921746e62bb229880774a311bfebfcf7579 Mon Sep 17 00:00:00 2001
From: Daniel Corbett 
Date: Thu, 17 May 2018 13:17:54 -0400
Subject: [PATCH] BUG/MEDIUM: stick-tables: Decrement ref_cnt in table_*
 converters

When using table_* converters ref_cnt was incremented
and never decremented causing entries to not expire.

The root cause appears to be that stktable_lookup_key()
was called within all sample_conv_table_* functions which was
incrementing ref_cnt and not decrementing after completion.

Added stktable_release() to the end of each sample_conv_table_*
function.

This should be backported to 1.8
---
 src/stick_table.c | 19 +++
 1 file changed, 19 insertions(+)

diff --git a/src/stick_table.c b/src/stick_table.c
index 3e44747..f1ad347 100644
--- a/src/stick_table.c
+++ b/src/stick_table.c
@@ -912,6 +912,7 @@ static int sample_conv_table_bytes_in_rate(const struct arg *arg_p, struct sampl
 
 	smp->data.u.sint = read_freq_ctr_period(&stktable_data_cast(ptr, bytes_in_rate),
 	   t->data_arg[STKTABLE_DT_BYTES_IN_RATE].u);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -948,6 +949,7 @@ static int sample_conv_table_conn_cnt(const struct arg *arg_p, struct sample *sm
 		return 0; /* parameter not stored */
 
 	smp->data.u.sint = stktable_data_cast(ptr, conn_cnt);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -984,6 +986,7 @@ static int sample_conv_table_conn_cur(const struct arg *arg_p, struct sample *sm
 		return 0; /* parameter not stored */
 
 	smp->data.u.sint = stktable_data_cast(ptr, conn_cur);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1021,6 +1024,7 @@ static int sample_conv_table_conn_rate(const struct arg *arg_p, struct sample *s
 
 	smp->data.u.sint = read_freq_ctr_period(&stktable_data_cast(ptr, conn_rate),
 	   t->data_arg[STKTABLE_DT_CONN_RATE].u);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1058,6 +1062,7 @@ static int sample_conv_table_bytes_out_rate(const struct arg *arg_p, struct samp
 
 	smp->data.u.sint = read_freq_ctr_period(&stktable_data_cast(ptr, bytes_out_rate),
 	   t->data_arg[STKTABLE_DT_BYTES_OUT_RATE].u);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1094,6 +1099,7 @@ static int sample_conv_table_gpt0(const struct arg *arg_p, struct sample *smp, v
 		return 0; /* parameter not stored */
 
 	smp->data.u.sint = stktable_data_cast(ptr, gpt0);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1130,6 +1136,7 @@ static int sample_conv_table_gpc0(const struct arg *arg_p, struct sample *smp, v
 		return 0; /* parameter not stored */
 
 	smp->data.u.sint = stktable_data_cast(ptr, gpc0);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1167,6 +1174,7 @@ static int sample_conv_table_gpc0_rate(const struct arg *arg_p, struct sample *s
 
 	smp->data.u.sint = read_freq_ctr_period(&stktable_data_cast(ptr, gpc0_rate),
 	  t->data_arg[STKTABLE_DT_GPC0_RATE].u);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1203,6 +1211,7 @@ static int sample_conv_table_gpc1(const struct arg *arg_p, struct sample *smp, v
 		return 0; /* parameter not stored */
 
 	smp->data.u.sint = stktable_data_cast(ptr, gpc1);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1240,6 +1249,7 @@ static int sample_conv_table_gpc1_rate(const struct arg *arg_p, struct sample *s
 
 	smp->data.u.sint = read_freq_ctr_period(&stktable_data_cast(ptr, gpc1_rate),
 	  t->data_arg[STKTABLE_DT_GPC1_RATE].u);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1276,6 +1286,7 @@ static int sample_conv_table_http_err_cnt(const struct arg *arg_p, struct sample
 		return 0; /* parameter not stored */
 
 	smp->data.u.sint = stktable_data_cast(ptr, http_err_cnt);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1313,6 +1324,7 @@ static int sample_conv_table_http_err_rate(const struct arg *arg_p, struct sampl
 
 	smp->data.u.sint = read_freq_ctr_period(&stktable_data_cast(ptr, http_err_rate),
 	   t->data_arg[STKTABLE_DT_HTTP_ERR_RATE].u);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1349,6 +1361,7 @@ static int sample_conv_table_http_req_cnt(const struct arg *arg_p, struct sample
 		return 0; /* parameter not stored */
 
 	smp->data.u.sint = stktable_data_cast(ptr, http_req_cnt);
+	stktable_release(t, ts);
 	return 1;
 }
 
@@ -1386,6 +1399,7 @@ static int sample_conv_table_http_req_rate(const struct arg *arg_p, struct sampl
 
 	smp->data.u.sint = read_freq_ctr_period(&stktable_data_cast(ptr, http_req_rate),
 	   t->data_arg[STKTABLE_DT_HTTP_REQ_RATE].u);
+	stktable_release(t, ts);
 	return 1;
 

Re: [PATCH] Make sure all the pollers get fd updates

[Olivier Houchard]

Hi,

On Fri, May 04, 2018 at 05:32:24PM +0200, Olivier Houchard wrote:
> Hi,
> 
> When the code was changed to use one poller per thread, we overlooked the
> fact that some fds can be shared between multiple threads, and when one
> event occured, that required the fd to be added or removed from the poller,
> the information would be sent only to the current thread, the other threads
> may totally miss it, and thus either miss events, or report spurious events.
> The attached patches is an attempt at fixing this, by adding a new global
> update list, in addition to the local thread update list.
> 
> This can't be applied to 1.8, as it uses code that was not, and probably won't
> be, backported, so a different patch, similar in spirit, will be developed.
> 

Here is a patch that should do the same for 1.8.

Regards,

Olivier
>From 3f8fe65b4433f4f8c543ff9b11c48554fe862f45 Mon Sep 17 00:00:00 2001
From: Olivier Houchard 
Date: Thu, 17 May 2018 18:34:02 +0200
Subject: [PATCH] BUG/MEDIUM: pollers: Use a global list for fd shared between
 threads.

With the old model, any fd shared by multiple threads, such as listeners
or dns sockets, would only be updated on one threads, so that could lead
to missed event, or spurious wakeups.
To avoid this, add a global list for fd that are shared, and only remove
entries from this list when every thread as updated its poller.
This subtly changes the semantics of updt_fd_polling(), as it now unlocks
the FD_LOCK on exit.

This is similar in spirit to commit 6b96f7289c2f401deef4bdc6e20792360807dde4
(with the bugfix from c55b88ece616afe0b28dc81eb39bad37b5f9c33f) applied,
but had to be rewrote, because of the differences between 1.8 and master.

This should only be applied to 1.8.
---
 include/common/hathreads.h |   4 ++
 include/proto/fd.h | 130 ++---
 include/types/fd.h |  13 +
 src/ev_epoll.c |  90 +--
 src/ev_kqueue.c|  83 +
 src/ev_poll.c  |  45 
 src/ev_select.c|  37 +
 src/fd.c   |   6 +++
 src/hathreads.c|   2 +-
 9 files changed, 327 insertions(+), 83 deletions(-)

diff --git a/include/common/hathreads.h b/include/common/hathreads.h
index 325a869a..86db4d5c 100644
--- a/include/common/hathreads.h
+++ b/include/common/hathreads.h
@@ -201,6 +201,8 @@ void thread_exit_sync(void);
 int  thread_no_sync(void);
 int  thread_need_sync(void);
 
+extern unsigned long all_threads_mask;
+
 #if defined(DEBUG_THREAD) || defined(DEBUG_FULL)
 
 /* WARNING!!! if you update this enum, please also keep lock_label() up to 
date below */
@@ -209,6 +211,7 @@ enum lock_label {
FDTAB_LOCK,
FDCACHE_LOCK,
FD_LOCK,
+   FD_UPDATE_LOCK,
POLL_LOCK,
TASK_RQ_LOCK,
TASK_WQ_LOCK,
@@ -330,6 +333,7 @@ static inline const char *lock_label(enum lock_label label)
case FDCACHE_LOCK: return "FDCACHE";
case FD_LOCK:  return "FD";
case FDTAB_LOCK:   return "FDTAB";
+   case FD_UPDATE_LOCK:   return "FD_UPDATE";
case POLL_LOCK:return "POLL";
case TASK_RQ_LOCK: return "TASK_RQ";
case TASK_WQ_LOCK: return "TASK_WQ";
diff --git a/include/proto/fd.h b/include/proto/fd.h
index bb91bb2c..b6199ccf 100644
--- a/include/proto/fd.h
+++ b/include/proto/fd.h
@@ -43,6 +43,9 @@ extern THREAD_LOCAL int fd_nbupdt; // number of updates in 
the list
 __decl_hathreads(extern HA_SPINLOCK_T __attribute__((aligned(64))) 
fdtab_lock);  /* global lock to protect fdtab array */
 __decl_hathreads(extern HA_RWLOCK_T   __attribute__((aligned(64))) 
fdcache_lock);/* global lock to protect fd_cache array */
 __decl_hathreads(extern HA_SPINLOCK_T __attribute__((aligned(64))) poll_lock); 
  /* global lock to protect poll info */
+__decl_hathreads(extern HA_SPINLOCK_T __attribute__((aligned(64))) 
fd_updt_lock); /* global lock to protect the update list */
+
+extern struct fdlist update_list; // Global update list
 
 /* Deletes an FD from the fdsets, and recomputes the maxfd limit.
  * The file descriptor is also closed.
@@ -96,14 +99,70 @@ void fd_process_cached_events();
 
 /* Mark fd  as updated for polling and allocate an entry in the update list
  * for this if it was not already there. This can be done at any time.
+ * This function expects the FD lock to be locked, and returns with the
+ * FD lock unlocked.
  */
 static inline void updt_fd_polling(const int fd)
 {
-   if (fdtab[fd].update_mask & tid_bit)
+   if ((fdtab[fd].update_mask & fdtab[fd].thread_mask) ==
+   fdtab[fd].thread_mask) {
+   HA_SPIN_UNLOCK(FD_LOCK, &fdtab[fd].lock);
/* already scheduled for update */
return;
-   fdtab[fd].update_mask |= tid_bit;
-   fd_updt[fd_nbupdt++] = fd;
+   }
+   if (fdtab[fd].thread_m

Rewrite image path based on HTTP_REQUEST

[Lotic Lists]

Hi experts

 

How can I rewrite a image path based on URL?

 

Example, users request the url www.example.com/images/logo.png, haproxy just
balance to backend servers normally.

Now users request www.newdomain.com, I need rewrite URI to
/images/new-logo.png

 

Tks.

Marcelo

 

Re: HAProxy multiple key type support - bug/feature (?) with DH parameters

[Arnaud Gavara]

Hello,

I allow myself to relaunch this email, can someone tell us if it's a bug or a 
configuration problem please?
I would like to use ECDSA certificates in addition to RSA but this problem is 
blocking me.

Regards,
Arnaud.


- Mail original -
> De: "Arnaud Gavara" 
> À: "haproxy" 
> Envoyé: Mercredi 2 Mai 2018 17:25:26
> Objet: Re: HAProxy multiple key type support - bug/feature (?) with DH 
> parameters

> Hello,
> 
> I resume this mail from Olivier because I think I meet the same problem.
> Like him, I need to use specific DH parameters. For this, I simply use the
> ability to add these DH parameters in the certificate file.
> These DH parameters are well taken into account if I specify the exact path of
> the certificate, for example:
> bind: 443 ssl crt certificate.pem.rsa
> 
> Then, I try to use the functionality described in the manual
> (https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crt) which
> allows to create a certificate bundle if we don't specify the explicit suffix
> in the configuration:
> bind: 443 ssl crt certificate.pem
> In this case, the certificate is well used (certificate.pem.rsa, same file) 
> but
> not its part containing the specific DH parameters. Indeed, if I do an SSL
> connection test (with testssl.sh for example), I observe that HAProxy uses its
> default DH parameters instead of using those of the file.
> 
> Of course, the goal is to be able to offer ECDSA certificates, but before 
> going
> to this step, I would have to use specific DH parameters.
> 
> Regards,
> Arnaud.
> 
> - Mail original -
>> De: "Olivier Doucet" 
>> À: "HAProxy" 
>> Envoyé: Vendredi 23 Mars 2018 15:58:27
>> Objet: HAProxy multiple key type support - bug/feature (?) with DH parameters
> 
>> Hello,
>> a few months ago I started using multiple key type support in HAProxy. It
>> means I have this in haproxy.cfg :
>> bind :443 ssl crt example.pem
>> 
>> And these files:
>> example.pem.rsa
>> example.pem.rsa.ocsp
>> example.pem.rsa.issuer
>> example.pem.ecdsa
>> example.pem.ecdsa.ocsp
>> example.pem.ecdsa.issuer
>> (see https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crt)
>> 
>> It is working very well :)
>> 
>> I now need to handle specific DH parameters for a customer. Before, I used
>> to add a DH block in pem file and it was working ... But here, the block is
>> simply ignored, despite what is said in config :
>> https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.2-tune.ssl.default-dh-param
>> "This value is not used if static Diffie-Hellman parameters are supplied
>> either directly in the certificate file or by using the ssl-dh-param-file
>> parameter"
>> 
>> I can confirm this behaviour happens only when certificate are loaded with
>> .rsa / .ecdsa extension : it is working if I rename example.pem.rsa to
>> example.pem
>> 
>> I tried to create a file example.pem.rsa.dh or example.pem.rsa.dhparam with
>> no luck (just tried those file names randomly :p).
>> 
>> Olivier
> 
> --
> Université de Montpellier
> Direction du Système d'Information et du Numérique
> Service des Moyens Informatiques
> Bureau réseaux, sécurité et téléphonie IP

-- 
Université de Montpellier
Direction du Système d'Information et du Numérique
Service des Moyens Informatiques
Bureau réseaux, sécurité et téléphonie IP