Hi,

HAProxy 1.9-dev9 was released on 2018/12/02. It added 147 new commits
after version 1.9-dev8.

This version will give some of us a bit of relief. It is the first one in
one year which finally integrates all the ongoing parallel development! I
thought about calling it -rc1 but I remembered that mixing -rc and -dev
causes some confusion, so it's dev9.

So we're starting to see the light at the end of the tunnel, which marks
the hopefully soon completion of something that started really bad a
year ago when we started to modify multiple interdependent areas at the
same time, leaving each developer with limited ways to test their code
in target situation. Just to illustrate, HTX development needed idle
connections rework to get keep-alive, which required the connection
management to be replaced, which in turn couldn't be tested without a
multi-stream outgoing mux (H2), which itself couldn't be implemented
before HTX... Now the loop is closed!

It doesn't mean it works well. Oh no, rest assured it's still full of
awful bugs! But for the first time we'll be able to focus on integration
bugs instead of focusing on bugs caused by temporary code, to experience
integration failures later as we've been doing for one year.

So from now on, we'll focus on testing, bug fixing, code cleanups,
polishing, and documentation. No new big stuff is expected anymore. In the
worst case we'll revert certain things if they look too broken and appear
unfixable (let's hope it will not be the case).

I'm quite hopeful in the next steps, having seen how all the infrastructure
changes managed over time to much better isolate bugs, resulting in their
diagnostic and fixing to be significantly faster than in the past.

Among the new stuff that we finally managed to integrate with this series,
I can name :
  - near completion of HTX porting : everything is supported, even
    redirects/http-rules/errorfiles, except Lua and cache (which may or
    may not be addressed soon, I've heard that the showstopper was the
    filters API but it's been addressed now). HTX opens new possibilities
    by remains in experimental status at release date. Ideally for 2.0
    we should have same coverage for legacy and HTX modes so that we can
    remove the legacy mode for 2.1.

  - server-side connection multiplexing (needed for H2, H3 later, maybe
    one day FCGI, who knows)

  - server-side connection pooling : idle connections don't quit anymore
    if the front connection vanishes, they can remain on the server to be
    used by another one. Expect a few minor adjustments on this part soon.

  - end-to-end H2 over HTX : by using the internal native HTTP representation
    (HTX), it's possible to transport H2 messages between both ends without
    denaturating the message by converting it to HTTP/1 like in the past.
    Since HTTP/1 doesn't allow to transport all H2 details, server-side H2
    only works when enabling HTX. Trailers are still not supported for now,
    so no gRPC yet (we'll check that for 2.0).

  - ALPN-based server-side protocol selection : haproxy can connect to a
    server, negotiate the protocols using ALPN (or NPN), then decide whether
    to transcode the HTX message to H1 or H2 depending on what is supported.

  - And Pieter's run-regtest script, which we'll abuse a lot very soon!

Now let's be honest, there are still quite some bugs. There's something
broken with Lua and cosockets related to recent stream-interfaces updates,
that I need to dig into much closer. The H2 mux over HTX is limited to 1GB
transfers in each direction, indicating that there's definitely something
I misunderstood about the way to transfer data over HTX. But now that
everything got integrated together it'll be much easier to walk through
the whole stack to address this!

Overall, and despite all the remaining FIXME tags, the code is getting
cleaner, much better layered, and easier to dig into for newcomers, which
is a critical aspect for the project's scalability.

We'll spend the whole week addressing pending issues and fixing newly
discovered ones. And quite frankly, I'm already convinced it will be the
same for the week after. I'd like to target next week-end as the moment
to migrate haproxy.org servers to 1.9, it will be a good indicator of the
reliability ;-)

As a side note, I'm noticing that we have not seen a major thread bug for
a while now, including on 1.9 which is far trickier than 1.8 on this aspect,
this shows that with a bit of patience, even complex stuff eventually
becomes rock solid!

It was very important for me to get all the foundations in place to prepare
the work for a cooler development period for 2.0. I think that some of the
technical doc will be merged after the 1.9 release and backported, since
it's mostly for developers and not as urgent as bug fixing.

I should soon create a "next" branch for the complex stuff that we'll
schedule for 2.1 and above, so that we don't reproduce the nightmare we've
known with massively conflicting technical updates. This model works very
well for the Linux kernel and allows integration issues to be detected very
early and not on Saturday while trying to merge, resulting in 20 unexpected
hours of work each time, plus the introduction of new bugs. Maybe we'll see
that after releasing 2.0 we'll simply merge next into master and continue
to work in parallel at different paces depending on the complexity of the
subjects. At least I hope so, it will then remove the pressure we're facing
on each release and will significantly improve the overall quality at the
release date.

Please find the usual URLs below :
   Site index       : http://www.haproxy.org/
   Discourse        : http://discourse.haproxy.org/
   Sources          : http://www.haproxy.org/download/1.9/src/
   Git repository   : http://git.haproxy.org/git/haproxy.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy.git
   Changelog        : http://www.haproxy.org/download/1.9/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
Baptiste Assmann (1):
      BUG/MINOR: ssl: ssl_sock_parse_clienthello ignores session id

Christopher Faulet (41):
      BUG/MINOR: proto_htx: only mark connections private if NTLM is detected
      BUG/MINOR: cfgparse: Fix transition between 2 sections with the same name
      BUG/MINOR: http: Use out buffer instead of trash to display error snapshot
      BUG/MINOR: htx: Fix block size calculation when a start-line is 
added/replaced
      BUG/MINOR: mux-h1: Fix processing of "Connection: " header on outgoing 
messages
      BUG/MEDIUM: mux-h1: Reset the H1 parser when an outgoing message is 
processed
      BUG/MINOR: proto_htx: Send outgoing data to client to start response 
processing
      BUG/MINOR: htx: Stop a header or a start line lookup on the first EOH or 
EOM
      MEDIUM: conn_stream: Add a way to get mux's info on a CS from the upper 
layer
      MINOR: mux-h1: Implement get_cs_info() callback
      MINOR: stream: Rely on CS's info if it exists and fallback on session's 
ones
      MINOR: proto_htx: Use conn_stream's info to set t_idle duration when 
possible
      MINOR: mux-h1: Don't rely on the stream anymore in h1_set_srv_conn_mode()
      MINOR: mux-h1: Write last chunk and trailers if not found in the HTX 
message
      MINOR: mux-h1: Be prepare to fail when EOM is added during trailers 
parsing
      MINOR: mux-h1: Subscribe to send in h1_snd_buf() when not all data have 
been sent
      MINOR: mux-h1: Consume channel's data in a loop in h1_snd_buf()
      MEDIUM: mux-h1: Add keep-alive outgoing connections in connections list
      MINOR: htx: Add function to add an HTX block just before another one
      MINOR: htx: Add function to iterate on an HTX message using HTX blocks
      MINOR: htx: Add a function to find the HTX block corresponding to a data 
offset
      MINOR: stats: Don't add end-of-data marker and trailers in the HTX 
response
      MEDIUM: htx: Change htx_sl to be a struct instead of an union
      MINOR: htx: Add the start-line offset for the HTX message in the HTX 
structure
      MEDIUM: htx: Don't rely on h1_sl anymore except during H1 header parsing
      MINOR: proto-htx: Use the start-line flags to set the HTTP messsage ones
      MINOR: htx: Add BODYLESS flags on the HTX start-line and the HTTP message
      MINOR: proto_htx: Use full HTX messages to send 100-Continue responses
      MINOR: proto_htx: Use full HTX messages to send 103-Early-Hints responses
      MINOR: proto_htx: Use full HTX messages to send 401 and 407 responses
      MINOR: proto_htx: Send valid HTX message when redir mode is enabled on a 
server
      MINOR: proto_htx: Send valid HTX message to send 30x responses
      MEDIUM: proto_htx: Convert all HTTP error messages into HTX
      MINOR: mux-h1: Process conn_mode on the EOH when no connection header is 
found
      MINOR: mux-h1: Change client conn_mode on an explicit close for the 
response
      MINOR: mux-h1: Capture bad H1 messages
      MAJOR: filters: Adapt filters API to be compatible with the HTX 
represenation
      MEDIUM: proto_htx/filters: Add data filtering during the forwarding
      MINOR: flt_trace: Adapt to be compatible with the HTX representation
      MEDIUM: compression: Adapt to be compatible with the HTX representation
      BUG/MINOR: cfgparse: Fix the call to post parser of the last sections 
parsed

Frédéric Lécaille (3):
      REGTEST: Add a basic test for the cache.
      REGTEST: Fix several issues.
      REGTEST: Fix LEVEL 4 script 0 of "connection" module.

Joseph Herlant (22):
      CLEANUP: fix typos in the htx subsystem
      CLEANUP: Fix typo in the chunk headers file
      CLEANUP: Fix typos in the h1 subsystem
      CLEANUP: Fix typos in the h2 subsystem
      CLEANUP: Fix a typo in the mini-clist header
      CLEANUP: Fix a typo in the proto_htx subsystem
      CLEANUP: Fix typos in the proto_tcp subsystem
      CLEANUP: Fix a typo in the signal subsystem
      CLEANUP: Fix a typo in the session subsystem
      CLEANUP: Fix a typo in the queue subsystem
      CLEANUP: Fix typos in the shctx subsystem
      CLEANUP: Fix typos in the socket pair protocol subsystem
      CLEANUP: Fix typos in the map management functions
      CLEANUP: Fix typo in the fwrr subsystem
      CLEANUP: Fix typos in the cli subsystem
      CLEANUP: Fix typo in the 51d subsystem
      CLEANUP: Fix a typo in the base64 subsystem
      CLEANUP: Fix a typo in the connection subsystem
      CLEANUP: Fix a typo in the protocol header file
      CLEANUP: Fix a typo in the checks header file
      CLEANUP: Fix typos in the file descriptor subsystem
      CLEANUP: Fix a typo in the listener subsystem

Lukas Tribus (2):
      BUILD/MINOR: ssl: fix build with non-alpn/non-npn libssl
      MINOR: ssl: free ctx when libssl doesn't support NPN

Olivier Houchard (25):
      BUILD: Makefile: Disable -Wcast-function-type if it exists.
      BUG/MEDIUM: h2: Don't bogusly error if the previous stream was closed.
      BUG/MEDIUM: mux_pt: Don't forget to unsubscribe() on attach.
      BUG/MEDIUM: connections: Wake the stream once the mux is chosen.
      BUG/MEDIUM: connections: Don't forget to detach the connection from the 
SI.
      BUG/MEDIUM: stream_interface: Don't check if the handshake is done.
      BUG/MEDIUM: stream_interface: Make sure we read all the data available.
      BUG/MEDIUM: h2: Call h2_process() if there's an error on the connection.
      BUG/MEDIUM: session: Remove the session from the session_list in 
session_free.
      BUG/MEDIUM: streams: Don't assume we have a CS in sess_update_st_con_tcp.
      BUG/MEDIUM: connections: Don't assume we have a mux in connect_server().
      BUG/MEDIUM: connections: Remove the connection from the idle list before 
destroy.
      BUG/MEDIUM: session: properly clean the outgoing connection before 
freeing.
      BUG/MEDIUM: mux_pt: Don't try to send if handshake is not done.
      MEDIUM: connections: Put H2 connections in the idle list if http-reuse 
always.
      MEDIUM: h2: Destroy a connection with no stream if it has no owner.
      MAJOR: sessions: Store multiple outgoing connections in the session.
      MEDIUM: session: Steal owner-less connections on end of transaction.
      MEDIUM: server: Be smarter about deciding to reuse the last server.
      BUG/MEDIUM: Special-case http_proxy when dealing with outgoing 
connections.
      MEDIUM: mux-h2: Implement h2_attach().
      MEDIUM: mux-h2: Don't bother flagging outgoing connections as TOOMANY.
      MINOR: connection: Fix a comment.
      MINOR: mux: add a "max_streams" method.
      MEDIUM: servers: Add a way to keep idle connections alive.

PiBa-NL (2):
      REGTEST/MINOR: script: add run-regtests.sh script
      REGTEST: lua: check socket functionality from a lua-task

Tim Duesterhus (2):
      BUG/MINOR: mworker: Do not attempt to close(2) fd -1
      CLEANUP: http: Fix typo in init_http's comment

William Lallemand (4):
      BUG/MEDIUM: mworker: fix FD leak upon reload
      BUG/MEDIUM: mworker: avoid leak of client socket
      REORG: mworker: declare master variable in global.h
      BUG/MEDIUM: listeners: CLOEXEC flag is not correctly set

Willy Tarreau (45):
      BUILD: compression: fix build error with DEFAULT_MAXZLIBMEM
      MINOR: compression: always create the compression pool
      BUILD: htx: fix fprintf format inconsistency on 32-bit platforms
      BUILD: buffers: buf.h requires unistd to get ssize_t on libmusl
      MINOR: initcall: introduce a way to register init functions to call at 
boot
      MINOR: init: process all initcalls in order at boot time
      MEDIUM: init: convert all trivial registration calls to initcalls
      MINOR: thread: provide a set of lock initialisers
      MINOR: threads: add new macros to declare self-initializing locks
      MEDIUM: init: use self-initializing spinlocks and rwlocks
      MINOR: initcall: apply initcall to all register_build_opts() calls
      MINOR: initcall: use initcalls for most post_{check,deinit} and 
per_thread*
      MINOR: initcall: use initcalls for section parsers
      MINOR: memory: add a callback function to create a pool
      MEDIUM: init: use initcall for all fixed size pool creations
      MEDIUM: memory: use pool_destroy_all() to destroy all pools on deinit()
      MEDIUM: initcall: use initcalls for a few initialization functions
      MEDIUM: memory: make the pool cache an array and not a thread_local
      MINOR: h2: make struct h2_ops static
      BUG/MINOR: connection: report mux modes when HTX is supported
      MINOR: htx: add a function to cut the beginning of a DATA block
      MINOR: h2: implement H2->HTX request header frame transcoding
      MEDIUM: mux-h2: register mux for both HTTP and HTX modes
      MEDIUM: mux-h2: make h2_rcv_buf() support HTX transfers
      MEDIUM: mux-h2: make h2_snd_buf() HTX-aware
      MEDIUM: mux-h2: add basic H2->HTX transcoding support for headers
      MEDIUM: mux-h2: implement emission of H2 headers frames from HTX blocks
      MEDIUM: mux-h2: implement the emission of DATA frames from HTX DATA blocks
      MEDIUM: mux-h2: support passing H2 DATA frames to HTX blocks
      BUG/MEDIUM: mux-h2: don't lose the first response header in HTX mode
      BUG/MEDIUM: mux-h2: remove the HTX EOM block on H2 response headers
      MINOR: listener: the mux_proto entry in the bind_conf is const
      MINOR: connection: create conn_get_best_mux_entry()
      MINOR: server: the mux_proto entry in the server is const
      MINOR: config: make sure to associate the proper mux to bind and servers
      MINOR: hpack: add ":path" to the list of common header fields
      MINOR: h2: add new functions to produce an HTX message from an H2 response
      MINOR: mux-h2: mention that the mux is compatible with both sides
      MINOR: mux-h2: implement an outgoing stream allocator : 
h2c_bck_stream_new()
      MEDIUM: mux-h2: start to create the outgoing mux
      MEDIUM: mux-h2: implement encoding of H2 request on the backend side
      MEDIUM: mux-h2: make h2_frt_decode_headers() direction-agnostic
      MEDIUM: mux-h2: make h2_process_demux() capable of processing responses 
as well
      BUG/MINOR: lb-map: fix unprotected update to server's score
      BUILD: threads: fix minor build warnings when threads are disabled

---

Reply via email to