HAProxy 1.9-dev2 was released on 2018/09/12. It added 199 new commits
after version 1.9-dev1.

Let's be clear about this one : it's mostly aimed at developers to resync
their ongoing work. We've changed a number of pretty sensitive stuff and
definitely expect to spot some interesting bugs starting with "this is
impossible" and ending with "I didn't remember we supported this". Warning
given :-)

Since 1.9-dev1, a number of additional changes were merged. This is always
a good sign, it indicates that the developers do not intend to significantly
modify their design anymore. We're still not close to a release though, but
the steady efforts are paying off.

Among the recent changes (forgive me if I forget anyone) that should be
user-visible, I can list :
  - removal of the synchronization point for a thinner rendez-vous point.
    This is used to propagate server status changes. It used to induce a
    huge CPU consumption on all threads when servers were changing very
    often. Now we don't need to wake the sleeping threads up anymore so
    the health checks are much less noticeable.

  - added support for the new keyword "proto" on "bind" and "server" lines.
    This permits to force the application level protocol (which normally is
    negociated via ALPN). We temporarily use it to develop the new HTTP/1
    mux and it is also useful to process clear-text HTTP/2. For example it
    is possible to strip TLS using front servers then forward this to a
    unique listener for HTTP processing.

  - the server queues used to be only ordered on request arrival time. Now
    they also support time priority offsets and classes. This means it is
    possible to decide among all the requests pending in a server queue
    which one will be picked first based on its priority. The classes are
    always considered first, which means that all requests in a low number
    class will be processed before any request in a high number class. Then
    the offsets can be enforced based on time, for example it's possible to
    enforce that requests matching this criterion will experience 250ms
    less queuing time than other requests. It can be useful to deliver
    javascript/CSS before images for example, or to boost premium level
    users compared to other ones.

  - a few new sample fetch functions were added to determine the number
    of available connections on a server or on a backend.

The rest is mostly infrastructure changes which are not (well should not
be) user-visible :
  - the logs can now be emitted without a stream. This will be required to
    emit logs from the muxes. The HTTP/2 mux now emits a few logs on some
    rare error cases.

  - make error captures work outside of a stream. The purpose is to be able
    to continue to capture bad messages at various protocol layers. Soon we
    should get the ability to capture invalid H2 frames and invalid HPACK
    sequences just like we used to do at the HTTP level. This will be needed
    to preserve this functionnality when HTTP/1 processing moves down to the
    mux. It could possibly even be used in other contexts (peers or SPOE
    maybe ?).

  - the error captures are now dynamically allocated. For now it remains
    simple but the purpose is to have the ability to keep only a limited
    number of captures (total and per proxy) in order to significantly
    reduce the memory usage for those with tens of thousands of backends,
    and at the same time to maintain several logs per proxy, and not just
    the last one for each direction.

  - the master-worker model has improve quite a bit. Now instead of using
    pipes between the master and the workers, it uses socket pairs. The
    goal here is to allow the master to better communicate with workers.
    More specifically, a second patch set introducing the support for
    communicating via socket pairs over UNIX sockets will be usable in
    conjunction with this, allowing the CLI to bounce from the master to
    the workers (not finished yet).

  - the HTTP definitions and semantics code were moved to a new file,
    http.c, which is version-agnostic. Thanks to this, proto_http.c now
    mostly deals with the analysers and sample processing. This is aimed
    at simplifying the porting of the existing code to the new model.

  - the new HTTP/1 parser that was implemented to convert HTTP/1 responses
    to HTTP/2 was completed so that it will be reusable by the upcoming
    HTTP/1 mux.

  - the connection scheduling was reworked to feature true async-I/O at
    every level : till now, when an upper layer wanted to send some data
    (let's say a response to a client), it had to enable data polling on
    the underlying conn_stream which itself requested polling of the FD
    in the fd_cache from the mux, resulting in this FD being seen as
    active and the send() callback to be called, and in turn the snd_buf()
    function of the data layer to be called to deliver the data to the
    lower level. It was an insane amount of round trips, especially for
    some protocols like H2 where the connection state is irrelevant to
    this. Now a direct snd_buf() attempt is performed, and if it fails,
    the caller is subscribed to the lower layer to be called when it
    becomes possible. This way only the required number of layers are
    crossed and woken up. It also allows to deliver more fine-grained
    information between them (e.g. use flags to filter on certain events).
    Due to the very long history of working in the previous way (since
    version 1.0), it would not be surprising if this change wakes up some
    long-burried zombie bugs. So any CLOSE_WAIT or CPU loop report would
    be welcome in this case.

There's also one initially announced feature which will probably not make
it in 1.9, it's the SSL certificate update from the CLI. After looking at
the impacts deeply with Emeric and William, we figured that the only way
to make it reliable and future-proof requires some non-trivial changes to
the way the certificates are currently indexed. It's not an enormous change
but one significant enough to require a full-time person for a few weeks,
so unless someone steps up and is autonomous enough on this one we'll have
to postpone. Likewise, the changes that William made to load certificates
and private keys from distinct files are supposedly mergeable but we have
to double-check before putting us into a dead-end.

Now the good news for those who have made the effort to read me till here
is that we'll increase the release rate to two releases per year. The
observation is that while the current model works reasonably fine, it's
extremely challenging to try to complete all the foundation changes before
starting to work on the promised features which entirely depend on them,
and once the release date approaches, we see pressure build up and heavy
conflicts starting to appear between various branches. Additionnally, we
all know that the vast majority of users test only after a release, causing
massive reports of issues at different layers just after the release. Last
but not least, distros which ship with a version tend to ship with an early
one which is not yet very stable, and this complicates their ability to
follow fixes.

So in order to improve the situation, we'll proceed differently : we'll
release a version around October-November like nowadays, and this version
will be focused on mostly technical stuff. Very often it will in fact
include some optmizations that were developed in the context of the
previous release but which were not dry enough to be merged. An example
of this could be the lockless fd_cache changes that were merged very early
in 1.9. And another release will happen around May for mostly functional
changes. These ones will be much more user-visible and will present a much
lower risk of regression. The May version will continue to be maintained
for a long time while the November version will be short-lived (around 1
year, mostly for advanced users).

As such, we'll release 1.9 as planned, around October-November depending
on how things go, and we'll emit a 2.0 around May with more user-visible
changes. 1.9 will stop getting fixes once 2.1 is released in November 2019,
but 2.0 will continue to be maintained. This means that distros should
definitely avoid to package odd releases (including 1.9) and should only
focus on even ones (starting with 2.0).

I also predict that with this improved model we'll start to see the
emergence of topic branches, which are merged once ready. It's will be less
problematic to skip a release because it will postpone a feature for only 6
months and not one year like now. This will cause less last-minute conflicts
between the various activities and should result in a better overall
stability in even versions, and a much smoother distributed development
cycle where it's even likely that different maintainers will be able to
emit development releases (specifically the technical ones which nobody
thinks about doing).

Now... back to the code ;-)


Please find the usual URLs below :
   Site index       : http://www.haproxy.org/
   Discourse        : http://discourse.haproxy.org/
   Sources          : http://www.haproxy.org/download/1.9/src/
   Git repository   : http://git.haproxy.org/git/haproxy.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy.git
   Changelog        : http://www.haproxy.org/download/1.9/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Complete changelog :
Baptiste Assmann (2):
      BUG/MEDIUM: dns/server: fix incomatibility between SRV resolution and 
server state file
      BUG/MINOR: dns: check and link servers' resolvers right after config 

Bertrand Jacquin (1):
      DOC: ssl: Use consistent naming for TLS protocols

Christopher Faulet (17):
      BUG/MINOR: buffers: Fix b_slow_realign when a buffer is realign without 
      MEDIUM: mux: Remove const on the buffer in mux->snd_buf()
      CLEANUP: backend: Move mux install to call it at only one place
      MINOR: conn_stream: add an tx buffer to the conn_stream
      MINOR: conn_stream: add cs_send() as a default snd_buf() function
      MINOR: backend: Try to find the best mux for outgoing connections
      MEDIUM: backend: don't rely on mux_pt_ops in connect_server()
      MINOR: mux: Add info about the supported side in alpn_mux_list structure
      MINOR: mux: Unlink ALPN and multiplexers to rather speak of mux protocols
      MINOR: mux: Print the list of existing mux protocols during HA startup
      BUG/MINOR: threads: Remove the unexisting lock label 
      BUG/MEDIUM: stream_int: Don't check CO_FL_SOCK_RD_SH flag to trigger cs 
      MINOR: mux: Change get_mux_proto to get an ist as parameter
      MINOR: mux: Improve the message with the list of existing mux protocols
      MINOR: mux/frontend: Add 'proto' keyword to force the mux protocol
      MINOR: mux/server: Add 'proto' keyword to force the multiplexer's protocol
      MEDIUM: mux: Use the mux protocol specified on bind/server lines

Cyril Bonté (2):
      BUG/MEDIUM: lua: socket timeouts are not applied
      BUG/MINOR: lua: fix extra 500ms added to socket timeouts

Emeric Brun (4):
      BUG/MINOR: ssl: empty connections reported as errors.
      BUG/MEDIUM: ssl: fix missing error loading a keytype cert from a bundle.
      BUG/MEDIUM: ssl: loading dh param from certifile causes unpredictable 
      BUG/MINOR: map: fix map_regm with backref

Emmanuel Hocdet (1):
      BUG/MEDIUM: ECC cert should work with TLS < v1.2 and openssl >= 1.1.1

Frédéric Lécaille (9):
      REGTEST/MINOR: Missing mandatory "ignore_unknown_macro".
      REGTEST/MINOR: Add a new class of regression testing files.
      BUG/MINOR: lua: Bad HTTP client request duration.
      REGEST/MINOR: Add reg testing files.
      REGTEST/MINOR: Add a reg testing file for b406b87 commit.
      BUG/MAJOR: thread: lua: Wrong SSL context initialization.
      REGTEST/MINOR: Add a reg testing file for 3e60b11.
      REGTEST/MINOR: lua: Add reg testing files for 70d318c.
      BUG/MINOR: server: Crash when setting FQDN via CLI.

Jens Bissinger (1):
      DOC: Fix spelling error in configuration doc

Lukas Tribus (1):
      DOC: dns: explain set server ... fqdn requires resolver

Olivier Houchard (39):
      MINOR: connections: Make rcv_buf mandatory and nuke cs_recv().
      MINOR: connections: Move rxbuf from the conn_stream to the h2s.
      MINOR: connections: Get rid of txbuf.
      MINOR: tasks: Allow tasklet_wakeup() to wakeup a task.
      MINOR: connections/mux: Add the wait reason(s) to wait_list.
      MINOR: stream_interface: Don't use si_cs_send() as a task handler.
      MINOR: stream_interface: Give stream_interface its own wait_list.
      MINOR: mux_h2: Don't use h2_send() as a callback.
      MINOR: checks: Add event_srv_chk_io().
      BUG/MEDIUM: tasks: Don't insert in the global rqueue if nbthread == 1
      BUG/MEDIUM: sessions: Don't use t->state.
      MINOR: tasks: Don't special-case when nbthreads == 1
      MINOR: fd cache: And the thread_mask with all_threads_mask.
      BUG/MEDIUM: streams: Don't forget to remove the si from the wait list.
      BUG/MEDIUM: tasklets: Add the thread as active when waking a tasklet.
      BUG/MEDIUM: stream-int: Check if the conn_stream exist in si_cs_io_cb.
      BUG/MEDIUM: H2: Activate polling after successful h2_snd_buf().
      BUG/MEDIUM: stream_interface: Call the wake callback after sending.
      BUG/MEDIUM: hlua: Make sure we drain the output buffer when done.
      MINOR: checks: Call wake_srv_chk() when we can finally send data.
      BUG/MEDIUM: stream_interface: try to call si_cs_send() earlier.
      BUG/MEDIUM: hlua: Don't call RESET_SAFE_LJMP if SET_SAFE_LJMP returns 0.
      MINOR: log: One const should be enough.
      BUG/MAJOR: kqueue: Don't reset the changes number by accident.
      BUG/MEDIUM: tasks: Don't forget to decrement task_list_size in 
      MEDIUM: connections: Don't reset the polling flags in conn_fd_handler().
      MEDIUM: connections/mux: Add a recv and a send+recv wait list.
      MEDIUM: connections: Get rid of the recv() method.
      MINOR: h2: Let user of h2_recv() and h2_send() know xfer has been done.
      MEDIUM: h2: always subscribe to receive if allowed.
      MEDIUM: h2: Don't use a wake() method anymore.
      MEDIUM: stream_interface: Make recv() subscribe when more data is needed.
      MINOR: connections: Add a "handle" field to wait_list.
      MEDIUM: mux_h2: Revamp the send path when blocking.
      MEDIUM: stream_interfaces: Starts receiving from the upper layers.
      MINOR: checks: Give checks their own wait_list.
      MINOR: conn_streams: Remove wait_list from conn_streams.
      BUG/MEDIUM: h2: Don't forget to empty the wait lists on destroy.
      BUG/MEDIUM: h2: Don't forget to set recv_wait_list to NULL in h2_detach.

Patrick Hemmer (9):
      MINOR: stream: rename {srv,prx}_queue_size to *_queue_pos
      MINOR: queue: store the queue index in the stream when enqueuing
      MINOR: queue: replace the linked list with a tree
      MEDIUM: add set-priority-class and set-priority-offset
      MEDIUM: queue: adjust position based on priority-class and priority-offset
      DOC: add documentation for prio_class and prio_offset sample fetches.
      BUG/MEDIUM: lua: reset lua transaction between http requests
      MINOR: add be_conn_free sample fetch
      MINOR: Add srv_conn_free sample fetch

William Lallemand (17):
      MEDIUM: mworker: remove register/unregister signal functions
      MEDIUM: mworker: use the haproxy poll loop
      BUG/MINOR: mworker: no need to stop peers for each proxy
      MINOR: mworker: mworker_cleanlisteners() delete the listeners
      MEDIUM: mworker: block SIGCHLD until the master is ready
      MEDIUM: mworker: never block SIG{TERM,INT} during reload
      MEDIUM: startup: unify signal init between daemon and mworker mode
      MINOR: mworker: don't deinit the poller fd when in wait mode
      MEDIUM: mworker: master wait mode use its own initialization
      MEDIUM: mworker: replace the master pipe by socketpairs
      MINOR: mworker: keep and clean the listeners
      MEDIUM: threads: close the thread-waker pipe during deinit
      MEDIUM: mworker: call per_thread deinit in mworker_reload()
      MEDIUM: protocol: use a custom AF_MAX to help protocol parser
      MEDIUM: protocol: sockpair protocol
      TESTS: add a python wrapper for sockpair@
      BUILD: fix build without thread

Willy Tarreau (96):
      BUG/MEDIUM: threads: fix the no-thread case after the change to the sync 
      BUG/MEDIUM: servers: check the queues once enabling a server
      BUG/MEDIUM: queue: prevent a backup server from draining the proxy's 
      MEDIUM: checks: use the new rendez-vous point to spread check result
      MEDIUM: haproxy: don't use sync_poll_loop() anymore in the main loop
      MINOR: threads: remove the previous synchronization point
      MAJOR: server: make server state changes synchronous again
      CLEANUP: server: remove the update list and the update lock
      BUG/MEDIUM: connection/mux: take care of serverless proxies
      MINOR: queue: make sure the pendconn is released before logging
      DOC: update the roadmap about priority queues
      DOC: update the layering design notes
      BUG/MEDIUM: server: update our local state before propagating changes
      BUG/MEDIUM: cli/threads: protect all "proxy" commands against concurrent 
      DOC: server/threads: document which functions need to be called 
with/without locks
      BUG/MEDIUM: cli/threads: protect some server commands against concurrent 
      BUG/MAJOR: queue/threads: make pendconn_redistribute not lock the server
      BUG/MEDIUM: connection: don't forget to always delete the list's head
      BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance 
      BUG/MEDIUM: check/threads: do not involve the rendez-vous point for 
status updates
      BUG/MINOR: chunks: do not store -1 into chunk_printf() in case of error
      BUG/MEDIUM: http: don't store exp_replace() result in the trash's length
      BUG/MEDIUM: http: don't store url_decode() result in the samples's length
      BUG/MEDIUM: dns: don't store dns_build_query() result in the trash's 
      BUG/MEDIUM: map: don't store exp_replace() result in the trash's length
      BUG/MEDIUM: connection: don't store recv() result into trash.data
      BUG/MEDIUM: cli/ssl: don't store base64dec() result in the trash's length
      MINOR: chunk: remove impossible tests on negative chunk->data
      MINOR: sample: remove impossible tests on negative smp->data.u.str.data
      BUG/MEDIUM: unix: provide a ->drain() function
      MINOR: connection: make conn_sock_drain() work for all socket families
      BUG/MEDIUM: mux_pt: dereference the connection with care in mux_pt_wake()
      MINOR: tools: make date2str_log() take some consts
      MINOR: thread: implement HA_ATOMIC_XADD()
      BUG/MINOR: stream: use atomic increments for the request counter
      BUG/MEDIUM: session: fix reporting of handshake processing time in the 
      BUG/MEDIUM: h2: fix risk of memory leak on malformated wrapped frames
      BUG/MAJOR: buffer: fix incorrect check in __b_putblk()
      MINOR: log: move the log code to sess_build_logline() to add extra 
      MINOR: log: make the backend fall back to the frontend when there's no 
      MINOR: log: make sess_build_logline() not dereference a NULL stream for 
      MINOR: log: don't unconditionally pick log info from s->logs
      CLEANUP: log: make the low_level lf_{ip,port,text,text_len} functions 
take consts
      MINOR: log: keep a copy of the backend connection early in 
      MINOR: log: do not dereference a null stream to access captures
      MINOR: log: be sure not to dereference a null stream for a target
      MINOR: log: don't check the stream-int's conn_retries if the stream is 
      MINOR: log: use NULL for the unique_id if there is no stream
      MINOR: log: keep a copy of s->flags early to avoid a dereference
      MINOR: log: use zero as the request counter if there is no stream
      MEDIUM: log: make sess_build_logline() support being called with no stream
      MINOR: log: provide a function to emit a log for a session
      MEDIUM: h2: produce some logs on early errors that prevent streams from 
being created
      BUG/MINOR: h1: fix buffer shift after realignment
      MINOR: connection: make the initialization more consistent
      MINOR: connection: add new function conn_get_proxy()
      MINOR: connection: add new function conn_is_back()
      BUG/MINOR: http/threads: atomically increment the error snapshot ID
      MINOR: snapshot: restart on the event ID and not the stream ID
      MINOR: snapshot: split the error snapshots into common and proto-specific 
      MEDIUM: snapshot: start to reorder the HTTP snapshot output a little bit
      MEDIUM: snapshot: implement a show() callback and use it for HTTP
      MINOR: proxy: add a new generic proxy_capture_error()
      MINOR: http: make the HTTP error capture rely on the generic proxy code
      MINOR: http: remove the pointer to the error snapshot in 
      REORG: cli: move the "show errors" handler from http to proxy
      BUG/MEDIUM: snapshot: take the proxy's lock while dumping errors
      MEDIUM: snapshots: dynamically allocate the snapshots
      MEDIUM: snapshot: merge the captured data after the descriptor
      REORG: http: move the HTTP semantics definitions to http.h/http.c
      REORG: http: move http_get_path() to http.c
      REORG: http: move error codes production and processing to http.c
      REORG: http: move the log encoding tables to log.c
      REORG: http: move some header value processing functions to http.c
      BUG/MINOR: h2: report asynchronous end of stream on closed connections
      REORG: h1: create a new h1m_state
      MINOR: h1: add the restart offsets into struct h1m
      MINOR: h1: remove the unused states from h1m_state
      MINOR: h1: provide a distinct init() function for request and response
      MINOR: h1: add a message flag to indicate that a message carries a 
      MINOR: h2: make sure h1m->err_pos field is correct on chunk error
      MINOR: h1: properly pre-initialize err_pos to -2
      MINOR: mux_h2: replace the req,res h1 messages with a single h1 message
      MINOR: h2: pre-initialize h1m->err_pos to -1 on the output path
      MEDIUM: h1: consider err_pos before deciding to accept a header name or 
      MEDIUM: h1: make the parser support a pointer to a start line
      MEDIUM: h1: let the caller pass the initial parser's state
      MINOR: h1: make the message parser support a null <hdr> argument
      MEDIUM: h1: support partial message parsing
      MEDIUM: h1: remove the useless H1_MSG_BODY state
      MINOR: h2: store the HTTP status into the H2S, not the H1M
      MINOR: h1: remove the HTTP status from the H1M struct
      MEDIUM: h1: implement the request parser as well
      MINOR: h1: add H1_MF_TOLOWER to decide when to turn header names to lower 
      MINOR: connection: pass the proxy when creating a connection
      BUG/MAJOR: h2: reset the parser's state on mux buffer full


Reply via email to