stable-bot: Bugfixes waiting for a release 2.2 (10), 2.1 (55), 2.0 (45)

[stable-bot]

Hi,

This is a friendly bot that watches fixes pending for the next haproxy-stable 
release!  One such e-mail is sent periodically once patches are waiting in the 
last maintenance branch, and an ideal release date is computed based on the 
severity of these fixes and their merge date.  Responses to this mail must be 
sent to the mailing list.


Last release 2.2.10 was issued on 2021-03-03.  There are currently 10 
patches in the queue cut down this way:
- 4 MEDIUM, first one merged on 2021-03-05
- 6 MINOR, first one merged on 2021-03-12

Thus the computed ideal release date for 2.2.11 would be 2021-04-04, which is 
in three weeks or less.

Last release 2.1.11 was issued on 2021-01-08.  There are currently 55 
patches in the queue cut down this way:
- 19 MEDIUM, first one merged on 2021-01-26
- 36 MINOR, first one merged on 2021-01-26

Thus the computed ideal release date for 2.1.12 would be 2021-03-23, which is 
in one week or less.

Last release 2.0.20 was issued on 2021-01-08.  There are currently 45 
patches in the queue cut down this way:
- 17 MEDIUM, first one merged on 2021-01-28
- 28 MINOR, first one merged on 2021-01-28

Thus the computed ideal release date for 2.0.21 would be 2021-03-25, which is 
in one week or less.

The current list of patches in the queue is:
 - 2.0, 2.1  - MEDIUM  : mux-h1: Always set CS_FL_EOI for 
response in MSG_DONE state
 - 2.0, 2.1  - MEDIUM  : spoe: Kill applets if there are 
pending connections and nbthread > 1
 - 2.0, 2.1  - MEDIUM  : config: don't pick unset values from 
last defaults section
 - 2.0, 2.1  - MEDIUM  : resolvers: Reset address for 
unresolved servers
 - 2.0, 2.1  - MEDIUM  : mux-h1: Fix handling of responses to 
CONNECT other than 200-ok
 - 2.0, 2.1  - MEDIUM  : ssl: check a connection's status 
before computing a handshake
 - 2.0, 2.1  - MEDIUM  : mux-h2: handle remaining read0 cases
 - 2.0, 2.1  - MEDIUM  : dns: Consider the fact that dns 
answers are case-insensitive
 - 2.0, 2.1  - MEDIUM  : mux-h2: do not quit the demux loop 
before setting END_REACHED
 - 2.1   - MEDIUM  : ssl/cli: abort ssl cert is freeing the 
old store
 - 2.0, 2.1  - MEDIUM  : stats: add missing INF_BUILD_INFO 
definition
 - 2.0, 2.1  - MEDIUM  : cli/shutdown sessions: make it 
thread-safe
 - 2.0, 2.1  - MEDIUM  : resolvers: Reset server address and 
port for obselete SRV records
 - 2.2   - MEDIUM  : session: NULL dereference possible 
when accessing the listener
 - 2.0   - MEDIUM  : mux-h2: Be sure to enter in demux loop 
even if dbuf is empty
 - 2.1, 2.2  - MEDIUM  : lists: Avoid an infinite loop in 
MT_LIST_TRY_ADDQ().
 - 2.0, 2.1  - MEDIUM  : proxy: use thread-safe stream killing 
on hard-stop
 - 2.0, 2.1  - MEDIUM  : filters/htx: Fix data forwarding when 
payload length is unknown
 - 2.0, 2.1  - MEDIUM  : vars: make functions 
vars_get_by_{name,desc} thread-safe
 - 2.0, 2.1  - MEDIUM  : mux-h2: fix read0 handling on partial 
frames
 - 2.1   - MEDIUM  : lists: Lock the element while we check 
if it is in a list.
 - 2.2   - MEDIUM  : filters: Set CF_FL_ANALYZE on channels 
when filters are attached
 - 2.2   - MEDIUM  : stick-tables: fix ref counter in table 
entry using multiple http tracksc.
 - 2.0, 2.1  - MINOR   : server: Init params before parsing a 
new server-state line
 - 2.0, 2.1  - MINOR   : server: Don't call fopen() with 
server-state filepath set to NULL
 - 2.0, 2.1  - MINOR   : peers: Wrong "new_conn" value for 
"show peers" CLI command.
 - 2.1   - MINOR   : mux_h2: missing space between "st" and 
".flg" in the "show fd" helper
 - 2.2   - MINOR   : ssl: don't truncate the file 
descriptor to 16 bits in debug mode
 - 2.0, 2.1  - MINOR   : stick-table: Always call 
smp_fetch_src() with a valid arg list
 - 2.1   - MINOR   : init: enforce strict-limits when using 
master-worker
 - 2.2   - MINOR   : hlua: Don't strip last non-LWS char in 
hlua_pushstrippedstring()
 - 2.0, 2.1  - MINOR   : http-ana: Only consider dst address to 
process originalto option
 - 2.2   - MINOR   : proxy/session: Be sure to have a 
listener to increment its counters
 - 2.1   - MINOR   : http-ana: Don't increment HTTP error 
counter on internal errors
 - 2.1   - MINOR   : mux-h2: Fix typo in scheme adjustment
 - 2.1   - MINOR   : stats: revert the change on ST_CONVDONE
 - 2.0, 2.1  - MINOR   : sample: check alloc_trash_c

Unexpectedly high memory usage when loading large maps

[Bren]

Hello,

(I tried subscribing to this list a few times but it appears that subscribing 
isn't working. Any responses will probably have to be CCed to me for now. Thank 
you.)

I've been testing adding a x-geoip-country header as shown in this blog post:

https://www.haproxy.com/blog/bot-protection-with-haproxy/

I generated the country_iso_code.map file which ended up being about 3.4 
million records and 64MB. It looks like this:

x.x.x.x/24 YY

Where x.x.x.x is the IP block and YY is the country code of course.

Then I loaded it into my config like this:

http-request set-header x-geoip-country 
%[src,map_ip(/usr/local/etc/haproxy/country_iso_code.map)]

By the way, the blog post appears to be incorrect here. It says to use map not 
map_ip which doesn't work:

http-request set-header x-geoip-country 
%[src,map(/etc/hapee-1.8/country_iso_code.map)]

Anyway, what shocked me is that haproxy now takes several seconds to load and 
uses about 860MB of memory. Without the "http-request set-header" line haproxy 
reloads nearly instantly and uses < 20MB.

This seems excessive to me but then I can imagine that this is due to haproxy 
loading the map data into memory in a way that provides super fast lookups at 
the expense of ram. I just wanted to reach out to verify that this is the case 
before pushing this into production. I couldn't find any mention anywhere on 
loading large maps into haproxy.

I'm using the haproxy:2.2-alpine Docker image to test this. Here's the stripped 
down production config I'm using:

# vim: syntax=haproxy
global
log stdout format raw local0
maxconn 2
ssl-default-bind-ciphers 
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers 
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
stats socket :8100
stats timeout 1h

defaults
balance roundrobin
backlog 1
log global
mode http

option contstats
option dontlog-normal
option dontlognull
option log-separate-errors
option httplog

timeout client 30s
timeout client-fin 30s
timeout connect 5s
timeout http-keep-alive 5s
timeout http-request 5s
timeout queue 10s
timeout server 30s
timeout tarpit 5s
timeout tunnel 30s

errorfile 400 /usr/local/etc/haproxy/errors/400.http
errorfile 403 /usr/local/etc/haproxy/errors/403.http
errorfile 408 /usr/local/etc/haproxy/errors/408.http
errorfile 500 /usr/local/etc/haproxy/errors/500.http
errorfile 502 /usr/local/etc/haproxy/errors/502.http
errorfile 503 /usr/local/etc/haproxy/errors/503.http
errorfile 504 /usr/local/etc/haproxy/errors/504.http

frontend fe-main
bind :8101 ssl crt /usr/local/etc/haproxy/server.pem alpn h2,http/1.1

maxconn 1
no option dontlog-normal
option http-buffer-request
option forwardfor
tcp-request inspect-delay 10s

capture request header Host len 64
capture request header X-Forwarded-For len 64
capture request header Referer len 256

http-request set-header x-geoip-country 
%[src,map_ip(/usr/local/etc/haproxy/country_iso_code.map)]

default_backend be-test

backend be-test
server web web:8200 maxconn 128


Any input on this would be much appreciated!

Thanks,
Bren

Re: [ANNOUNCE] haproxy-2.3.7

[Dmitry Sivachenko]

> On 16 Mar 2021, at 18:01, Christopher Faulet  wrote:
> 
> Hi,
> 
> HAProxy 2.3.7 was released on 2021/03/16. It added 62 new commits
> after version 2.3.6.
> 
> This release is mainly about two subjects : The fix of bugs into the
> resolvers part, mainly revealed since the last release and several
> scalability improvements backported from the development version.
> 

<...>

Hello,

among other things, this version also introduces new warning (clang version 
11.0.0):

src/mux_h2.c:4032:49: warning: implicit conversion from 'int' to 'unsigned 
short' changes value from -32769 to 32767 [-Wconstant-conversion]
HA_ATOMIC_AND(&h2c->wait_event.tasklet->state, ~TASK_F_USR1);
~~~^
include/haproxy/atomic.h:270:62: note: expanded from macro 'HA_ATOMIC_AND'
#define HA_ATOMIC_AND(val, flags)__atomic_and_fetch(val, flags, 
__ATOMIC_SEQ_CST)



src/mux_fcgi.c:3477:51: warning: implicit conversion from 'int' to 'unsigned 
short' changes value from -32769 to 32767 [-Wconstant-conversion]
HA_ATOMIC_AND(&fconn->wait_event.tasklet->state, ~TASK_F_USR1);
~^
include/haproxy/atomic.h:270:62: note: expanded from macro 'HA_ATOMIC_AND'
#define HA_ATOMIC_AND(val, flags)__atomic_and_fetch(val, flags, 
__ATOMIC_SEQ_CST)



src/mux_h1.c:2456:49: warning: implicit conversion from 'int' to 'unsigned 
short' changes value from -32769 to 32767 [-Wconstant-conversion]
HA_ATOMIC_AND(&h1c->wait_event.tasklet->state, ~TASK_F_USR1);
~~~^
include/haproxy/atomic.h:270:62: note: expanded from macro 'HA_ATOMIC_AND'
#define HA_ATOMIC_AND(val, flags)__atomic_and_fetch(val, flags, 
__ATOMIC_SEQ_CST)

[ANNOUNCE] haproxy-2.3.7

[Christopher Faulet]

Hi,

HAProxy 2.3.7 was released on 2021/03/16. It added 62 new commits
after version 2.3.6.

This release is mainly about two subjects : The fix of bugs into the
resolvers part, mainly revealed since the last release and several
scalability improvements backported from the development version.

In the 2.3.6, some fixes in the resolvers part about the cached records
expiration revealed design problems leading to pretty annoying bugs around
the SRV records. Servers based on SRV resolution might enter in a flapping
loop, switching them from UP state to DOWN and UP again at each
resolution. The current design has clearly shown its limitations and it is
already planned to refactor it. However, this will not take part of the
stable releases. Thus several patches were pushed to (let's hope so) fix all
known issues, based on the current design. First the detection of cached
additional records was fixed, just like the record expiration loop, fixing
this way the flapping servers bug. The triggering of extra DNS resolutions
to overcome the lack of some additional records in a SRV response was
improved, as the ability to stop them. And finally, the time unit for cached
records is now the millisecond. It should mitigate some edge cases with
short-live records and heavy resolution frequencies. Note that all of this
is also true for the 2.2 and a new release will be emitted soon, once this
one will have proved it reliability.

A few of the scalability fixes related to issues 922 and 958 that arrived
into 2.4-dev10 and 2.4-dev11 were now backported, as planned. These roughly
fall into 3 main categories :

  * architecture-specific atomic operations: support for LSE instructions on
armv8.1-a that has been in testing since November and found to be
necessary to avoid crashes on 16+ cores on arm64 machines

  * general locking cost reduction: some global lists were still heavily
updated (like streams, actconns, buffers), often causing excessive
contention and even rare panics under too many threads. These lists were
however never used in any particular order and could be were split by
thread.

  * SSL batching limits to tag heavy handshakes and avoid them excessively
delaying all other processing (mixed traffic is much smoother now)

While not as good as 2.4, this managed to substantially stabilize 2.3 for
large CPUs. After a few weeks, we'll also consider some of these patches for
2.2 so that one LTS version can work better on large machines.

Of course, as usual, there are also some other fixes here and there. Mainly :

 * The filters part was fixed to be sure the end analyzer (flt_end_analyse) is
   always called for the request and the response, especially when the request
   analysis is finished before the response start.

 * William fixed possible bugs about the listeners. Listeners are not
   necessarily present when the client is an applet (peers, spoe, Lua) and
   we need to be careful when updating counters. It was too hard to say
   whether those could be triggered but there was at least one way
   consisting in adding TCP rules to an SPOE backend.

 * Two bugs in the tcpchecks were fixed. The first one was about the
   agent-checks when mixed with the health-checks. When a agent was setting the
   server state, instead of updating the agent's .health threshold, the
   health-check's one was updated. Thus the agent was competing with the
   health-check and might mark a DOWN server as UP, while the precedence should
   be on the health-checks. The second bug was a double free error on invalid
   tcp-check/http-check rule.

 * Emeric fixed a ref counting bug into the stick-tables. Setting multiple
   http track-scX rules generated never expiring entries. This bug was
   introduced in the 2.2, when the http actions management was refactored.


Thanks to everyone for this release !


Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Discourse: http://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Wiki : https://github.com/haproxy/wiki/wiki
   Sources  : http://www.haproxy.org/download/2.3/src/
   Git repository   : http://git.haproxy.org/git/haproxy-2.3.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-2.3.git
   Changelog: http://www.haproxy.org/download/2.3/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/


---
Complete changelog :
Amaury Denoyelle (1):
  BUG/MINOR: backend: fix condition for reuse on mode HTTP

Baptiste Assmann (1):
  MINOR: resolvers: new function find_srvrq_answer_record()

Christopher Faulet (23):
  BUG/MINOR: hlua: Don't strip last non-LWS char in 
hlua_pushstrippedstring()
  BUG/MEDIUM: filters: Set CF_FL_ANALYZE on channels when filters are 
attached
  BUG/MINOR: tcpcheck: Update .health threshold of agent inside an 
agent-check
  BUG/MINOR: pro

Re: [2.2.9] 100% CPU usage

[Willy Tarreau]

On Tue, Mar 16, 2021 at 01:46:48PM +0100, Maciej Zdeb wrote:
> In the last message I said that the old process (after
> reload) is consuming cpu for lua processing and that's not true, it is
> processing other things also.
> 
> I'll take a break. ;) Then I'll verify if the issue exists on 2.3 and maybe
> 2.4 branch. For each version I need a week or two to be sure the issue does
> not occur. :(

Don't worry, we know how it is. On the developer side it's annoying to
have such bugs, but on the other hand, when bugs take one week to become
visible, it's likely that there aren't that many left, so this gives some
incentive to find the root cause :-)

The perf top you sent indicates that *something* is waking a task in
loops. Unfortunately we don't know what. In 2.4 we have more tools to
observe this, so if you still observe it I may even indicate you some
gdb commands or suggest a few patches to figure exactly what is
happening.

> If 2.3 and 2.4 behaves the same way the 2.2 does, I'll try to confirm if
> there is any relation between infinite loops and custom configuration:
> - lua scripts (mainly used for header generation/manipulation),
> - spoe (used for sending metadata about each request to external service),
> - peers (we have a cluster of 12 HAProxy servers connected to each other).

Any of the 3 could be indirectly responsible, as the stopping state
sometimes has to be taken care of in special ways. The peers are a bit
less likely because they're used a lot, though that doesn't mean they're
immune to this. But Lua and SPOE are a bit less commonly deployed and
could still be facing some rare stupid corner cases that have not yet
been identified. Obviously I have no idea which ones :-/

Cheers,
Willy

Re: [2.2.9] 100% CPU usage

[Maciej Zdeb]

Sorry for spam. In the last message I said that the old process (after
reload) is consuming cpu for lua processing and that's not true, it is
processing other things also.

I'll take a break. ;) Then I'll verify if the issue exists on 2.3 and maybe
2.4 branch. For each version I need a week or two to be sure the issue does
not occur. :(

If 2.3 and 2.4 behaves the same way the 2.2 does, I'll try to confirm if
there is any relation between infinite loops and custom configuration:
- lua scripts (mainly used for header generation/manipulation),
- spoe (used for sending metadata about each request to external service),
- peers (we have a cluster of 12 HAProxy servers connected to each other).

Kind regards,

wt., 16 mar 2021 o 13:05 Maciej Zdeb  napisał(a):

> Below is the output from perf top - it happens during reload (all threads
> on an old process spike and use 100% cpu, processing lua) and after 15-30
> seconds old process exits. It is probably a different bug, because it
> happens on an old process and I have no idea how it could affect the new
> process. Maybe through peers protocol? Still both issues are related to lua
> scripts (on my initial email in low_cpu_thread.txt it hung on lua script
> responsible for removing http headers on hlua_http_get_headers function).
>
> Samples: 770K of event 'cycles:ppp', Event count (approx.): 336817908209
> Overhead  Shared Object   Symbol
>   36.39%  haproxy [.]
> process_runnable_tasks
>   34.82%  haproxy [.]
> run_tasks_from_lists
>   10.04%  haproxy [.] __task_wakeup
>5.63%  libc-2.27.so[.]
> __memset_avx2_erms
>1.87%  haproxy [.] hlua_action
>0.79%  haproxy [.] hlua_ctx_init
>0.77%  haproxy [.]
> stream_release_buffers
>0.58%  libcrypto.so.1.1[.]
> rsaz_1024_sqr_avx2
>0.56%  haproxy [.]
> stream_int_notify
>0.49%  haproxy [.]
> task_run_applet
>0.42%  haproxy [.]
> hlua_sample_fetch_wrapper
>0.38%  haproxy [.]
> spoe_handle_appctx
>0.38%  [kernel][k] do_syscall_64
>0.37%  haproxy [.] eb32sc_insert
>0.32%  [vdso]  [.]
> __vdso_clock_gettime
>0.27%  haproxy [.]
> hlua_ctx_destroy
>0.22%  haproxy [.]
> stktable_get_entry
>0.20%  haproxy [.]
> process_stream
>0.19%  [kernel][k]
> audit_filter_rules.constprop.13
>0.17%  haproxy [.]
> si_applet_wake_cb
>0.14%  libcrypto.so.1.1[.]
> rsaz_1024_mul_avx2
>
> wt., 16 mar 2021 o 12:00 Maciej Zdeb  napisał(a):
>
>> Sure, patch from Christopher attached. :)
>>
>> wt., 16 mar 2021 o 10:58 Willy Tarreau  napisał(a):
>>
>>> Hi Maciej,
>>>
>>> On Tue, Mar 16, 2021 at 10:55:11AM +0100, Maciej Zdeb wrote:
>>> > Hi,
>>> >
>>> > I'm returning with bad news, the patch did not help and the issue
>>> occurred
>>> > today (on patched 2.2.10). It is definitely related to reloads,
>>> however it
>>> > is very rare issue it worked flawlessly the whole week.
>>>
>>> OK. Just as a reminder (because I can't find it here), can you please
>>> remind us what this patch was ?
>>>
>>> Thanks,
>>> Willy
>>>
>>

Re: [2.2.9] 100% CPU usage

[Maciej Zdeb]

Below is the output from perf top - it happens during reload (all threads
on an old process spike and use 100% cpu, processing lua) and after 15-30
seconds old process exits. It is probably a different bug, because it
happens on an old process and I have no idea how it could affect the new
process. Maybe through peers protocol? Still both issues are related to lua
scripts (on my initial email in low_cpu_thread.txt it hung on lua script
responsible for removing http headers on hlua_http_get_headers function).

Samples: 770K of event 'cycles:ppp', Event count (approx.): 336817908209
Overhead  Shared Object   Symbol
  36.39%  haproxy [.]
process_runnable_tasks
  34.82%  haproxy [.]
run_tasks_from_lists
  10.04%  haproxy [.] __task_wakeup
   5.63%  libc-2.27.so[.]
__memset_avx2_erms
   1.87%  haproxy [.] hlua_action
   0.79%  haproxy [.] hlua_ctx_init
   0.77%  haproxy [.]
stream_release_buffers
   0.58%  libcrypto.so.1.1[.]
rsaz_1024_sqr_avx2
   0.56%  haproxy [.]
stream_int_notify
   0.49%  haproxy [.]
task_run_applet
   0.42%  haproxy [.]
hlua_sample_fetch_wrapper
   0.38%  haproxy [.]
spoe_handle_appctx
   0.38%  [kernel][k] do_syscall_64
   0.37%  haproxy [.] eb32sc_insert
   0.32%  [vdso]  [.]
__vdso_clock_gettime
   0.27%  haproxy [.]
hlua_ctx_destroy
   0.22%  haproxy [.]
stktable_get_entry
   0.20%  haproxy [.] process_stream
   0.19%  [kernel][k]
audit_filter_rules.constprop.13
   0.17%  haproxy [.]
si_applet_wake_cb
   0.14%  libcrypto.so.1.1[.]
rsaz_1024_mul_avx2

wt., 16 mar 2021 o 12:00 Maciej Zdeb  napisał(a):

> Sure, patch from Christopher attached. :)
>
> wt., 16 mar 2021 o 10:58 Willy Tarreau  napisał(a):
>
>> Hi Maciej,
>>
>> On Tue, Mar 16, 2021 at 10:55:11AM +0100, Maciej Zdeb wrote:
>> > Hi,
>> >
>> > I'm returning with bad news, the patch did not help and the issue
>> occurred
>> > today (on patched 2.2.10). It is definitely related to reloads, however
>> it
>> > is very rare issue it worked flawlessly the whole week.
>>
>> OK. Just as a reminder (because I can't find it here), can you please
>> remind us what this patch was ?
>>
>> Thanks,
>> Willy
>>
>

Re: [2.2.9] 100% CPU usage

[Maciej Zdeb]

Sure, patch from Christopher attached. :)

wt., 16 mar 2021 o 10:58 Willy Tarreau  napisał(a):

> Hi Maciej,
>
> On Tue, Mar 16, 2021 at 10:55:11AM +0100, Maciej Zdeb wrote:
> > Hi,
> >
> > I'm returning with bad news, the patch did not help and the issue
> occurred
> > today (on patched 2.2.10). It is definitely related to reloads, however
> it
> > is very rare issue it worked flawlessly the whole week.
>
> OK. Just as a reminder (because I can't find it here), can you please
> remind us what this patch was ?
>
> Thanks,
> Willy
>


0001-BUG-MEDIUM-lists-Avoid-an-infinite-loop-in-MT_LIST_T.patch
Description: Binary data

Re: [2.2.9] 100% CPU usage

[Willy Tarreau]

Hi Maciej,

On Tue, Mar 16, 2021 at 10:55:11AM +0100, Maciej Zdeb wrote:
> Hi,
> 
> I'm returning with bad news, the patch did not help and the issue occurred
> today (on patched 2.2.10). It is definitely related to reloads, however it
> is very rare issue it worked flawlessly the whole week.

OK. Just as a reminder (because I can't find it here), can you please
remind us what this patch was ?

Thanks,
Willy

Re: [2.2.9] 100% CPU usage

[Maciej Zdeb]

Hi,

I'm returning with bad news, the patch did not help and the issue occurred
today (on patched 2.2.10). It is definitely related to reloads, however it
is very rare issue it worked flawlessly the whole week.

wt., 9 mar 2021 o 09:17 Willy Tarreau  napisał(a):

> On Tue, Mar 09, 2021 at 09:04:43AM +0100, Maciej Zdeb wrote:
> > Hi,
> >
> > After applying the patch, the issue did not occur, however I'm still not
> > sure it is fixed. Unfortunately I don't have a reliable way to trigger
> it.
>
> OK. If it's related, it's very possible that some of the issues we've
> identified there recently are at least responsible for a part of the
> problem. In short, if the CPUs are too fair, some contention can last
> long because two steps are required to complete such operations. We've
> added calls to cpu_relax() in 2.4 and some are queued in 2.3 already.
> After a while, some of them should also be backported to 2.2 as they
> significantly improved the situation with many threads.
>
> > pt., 5 mar 2021 o 22:07 Willy Tarreau  napisal(a):
> >
> > > Note, before 2.4, a single thread can execute Lua scripts at once,
> > > with the others waiting behind, and if the Lua load is important, maybe
> > > this can happen (but I've never experienced it yet, and the premption
> > > interval is short enough not to cause issues in theory).
> > >
> > I'm not sure if related but during every reload, for a couple seconds all
> > 12 threads on OLD pid are using 100% cpu and then one after one core
> return
> > to normal usage, finally the old haproxy process exits. I have no idea
> why
> > it is behaving like that.
>
> It could be related but it's hard to tell. It is also possible that
> for any reason the old threads constantly believe they have something
> to do, for example a health check that doesn't get stopped and that
> keep ringing.
>
> > > Maciej, if this happens often,
> > > would you be interested in running one machine on 2.4-dev11 ?
> >
> > It is a very rare issue and of course it occurs only in production
> > environment. :(
>
> Obviously!
>
> > I'm very willing to test the 2.4 version, especially with
> > that tasty lua optimization for multiple threads, but I can't do it on
> > production until it's stable.
>
> This makes sense. We'll try to issue 2.3 with some thread fixes this
> week, maybe it will be a possible step for you then.
>
> Cheers,
> Willy
>

Bid Writing, Fundraising and Volunteering Workshops

[NFP Workshops]

NFP WORKSHOPS
Affordable Training Courses

18 Blake Street, York YO1 8QG01133 280988




Bid Writing: The Basics


 Do you know the most common reasons for rejection? Are you gathering the right 
evidence? Are you making the right arguments? Are you using the right 
terminology? Are your numbers right? Are you learning from rejections? 

Are you assembling the right documents? Do you know how to create a clear and 
concise standard funding bid? Are you communicating with people or just 
excluding them? Do you know your own organisation well enough? 

Are you thinking through your projects carefully enough? Do you know enough 
about your competitors? Are you answering the questions funders will ask 
themselves about your application? Are you submitting applications correctly?
ONLINE VIA ZOOM
10.00 TO 12.30
COST £95.00
MON 22 MAR 2021
BOOKING LINK
MON 12 APR 2021
BOOKING LINK
MON 26 APR 2021
BOOKING LINK




Bid Writing: Advanced

 Are you applying to the right trusts? Are you applying to enough trusts? Are 
you asking for the right amount of money? Are you applying in the right ways? 
Are your projects the most fundable projects? 

Are you carrying out trust fundraising in a professional way? Are you 
delegating enough work? Are you highly productive or just very busy? Are you 
looking for trusts in all the right places? 

How do you compare with your competitors for funding? Is the rest of your 
fundraising hampering your bids to trusts? Do you understand what trusts are 
ideally looking for?
ONLINE VIA ZOOM
10.00 TO 12.30
COST £95.00
TUE 23 MAR 2021
BOOKING LINK
TUE 13 APR 2021
BOOKING LINK
TUE 27 APR 2021
BOOKING LINK



Major Donor Fundraising

 Major Donor Characteristics, Motivations and Requirements. Researching and 
Screening Major Donors. Encouraging, Involving and Retaining Major Donors.

Building Relationships with Major Donors. Major Donor Events and Activities. 
Setting Up Major Donor Clubs. Asking For Major Gifts. Looking After and 
Reporting Back to Major Donors.  
 
Delivering on Major Donor Expectations. Showing Your Appreciation to Major 
Donors. Fundraising Budgets and Committees.   
ONLINE VIA ZOOM
10.00 TO 12.30
COST £95
WED 14 APR 2021
BOOKING LINK
THU 10 JUN 2021
BOOKING LINK



Corporate Fundraising 

Who are these companies? Why do they get involved? What do they like? What can 
you get from them? What can you offer them? What are the differences between 
donations, sponsorship, advertising and cause related marketing? 

Are companies just like trusts? How do you find these companies? How do you 
research them? How do you contact them? How do you pitch to them? How do you 
negotiate with them? 

When should you say no? How do you draft contracts? How do you manage the 
relationships? What could go wrong? What are the tax issues? What are the legal 
considerations?
ONLINE VIA ZOOM
10.00 TO 12.30
COST £95
THU 29 APR 2021
BOOKING LINK
THU 24 JUN 2021
BOOKING LINK



Recruiting and Managing Volunteers
 Where do you find volunteers? How do you find the right volunteers? How do you 
attract volunteers? How do you run volunteer recruitment events? How do you 
interview volunteers? 

How do you train volunteers? How do you motivate volunteers? How do you involve 
volunteers? How do you recognise volunteers? How do you recognise problems with 
volunteers? How do you learn from volunteer problems? 

How do you retain volunteers? How do you manage volunteers? What about 
volunteers and your own staff? What about younger, older and employee 
volunteers?

ONLINE VIA ZOOM
10.00 TO 12.30
COST £95
THU 13 MAY 2021
BOOKING LINK



Legacy Fundraising 

Why do people make legacy gifts? What are the ethical issues? What are the 
regulations? What are the tax issues? What are the statistics? What are the 
trends? How can we integrate legacy fundraising into our other fundraising? 

What are the sources for research? How should we set a budget? How should we 
evaluate our results? How should we forecast likely income? Should we use 
consultants? How should we build a case for support? 

What media and marketing channels should we use? What about in memory giving? 
How should we setup our admin systems? What are the common problems & pitfalls?
ONLINE VIA ZOOM
10.00 TO 12.30
COST £95
THU 27 MAY 2021
BOOKING LINK



Community Fundraising 

What is Community Fundraising? How can we develop a community fundraising plan? 
Do we need a communications plan? How could we best manage all the data? Should 
we be planning ahead? How can we best encourage fundraising activities? What is 
the best way to organise events? How should we manage fundraising volunteers? 
How can we engage with community groups? Do we need to engage with local 
businesses?
ONLINE VIA ZOOM
10.00 TO 12.30
COST £95  
MON 15 MAR 2021
BOOKING LINK



Writing For Work & The Web 

Why care about words? How should I prepare? How can I create readable sections? 
Is it only me that suffers writer’s block? Am I really writing in plain 
English? 

I’m heartbroken

[Lexi Davis]

Hey,

I’ve tried to contact you a couple of times, but I feel as though I was
left on 'seen'.  That’s totally fine. Kind of.

I'm closing the door for now, but I'm leaving the window (of opportunity)
open.

Talk soon,

Lexi.


-Original Message-

Hi,

Just following up here.

Hope you had a chance to check out the Smartproxy blog

and see what kind of content we deliver.

Do you offer sponsored post opportunities on haproxy.org?

Many thanks,

Lexi.


-Original Message-

Hey,

Hope you’re doing well.

Last time I offered you a 700-800 word in-depth article that I wrote for
your audience.

The Smartproxy blog

receives 14K organic visitors each month. This must mean that we deliver
only high quality content for our partners. It would definitely help you to
boost your engagement.

Would you be interested to work together and publish it on haproxy.org?

Kindly,

Lexi


-Original Message-

Hi,

I’m Lexi, marketing manager at Smartproxy - A smart but down to earth
techhead platform that helps clients solve data access and entrepreneurship
problems.

I wanted to see if you’d be interested in featuring *unique & high quality*
content that our tech geeks wrote for your haproxy.org audience.

I’d like to offer you 2 article topics to choose from that would make your
website stand out.

What kind of content are you looking for at the moment? I'll do my best to
fit your needs – our team is always on stand by.

Best wishes,

Lexi.