Re: [ANNOUNCE] haproxy-2.0-dev6 + Conf reminder

[Willy Tarreau]

On Fri, Jun 07, 2019 at 10:55:18AM +0200, Luke Seelenbinder wrote:
> I just realized we have both referred to two different variants of the email, 
> one with an s and one without:
> 
> submissi...@haproxy.com and submiss...@haproxy.com

Ah sorry about that! Yes it's without the trailing "s" as I
first copy-pasted in the previous e-mail.

> I sent mine to submiss...@haproxy.com (the first one you linked).

Great!

> Maybe you want to clarify for everyone which is the canonical before all the
> submissions end up accidentally hitting a junk drawer somewhere? :-)

Yes you're right, thank you for notifying me. It seemed more natural
to me to have this trailing "s", reason why I made the mistake. Let's
hope I'm the only one making it.

Cheers,
Willy

Re: [ANNOUNCE] haproxy-2.0-dev6 + Conf reminder

[Luke Seelenbinder]

I just realized we have both referred to two different variants of the email, 
one with an s and one without:

submissi...@haproxy.com and submiss...@haproxy.com

I sent mine to submiss...@haproxy.com (the first one you linked).

Maybe you want to clarify for everyone which is the canonical before all the 
submissions end up accidentally hitting a junk drawer somewhere? :-)

—
Luke Seelenbinder
Stadia Maps | Founder
stadiamaps.com

> On Jun 7, 2019, at 10:14, Willy Tarreau  wrote:
> 
> On Fri, Jun 07, 2019 at 10:03:59AM +0200, Luke Seelenbinder wrote:
>>> On another point, I was told that very few proposals for talks at the
>>> HAProxyConf were sent to date. That doesn't surprize me and I haven't
>>> even sent mine yet either :-)  Please keep in mind that the deadline for
>>> proposals submissions is in two weeks and that time flies. So instead of
>>> wasting your time reading my lengthy boring e-mails, please take a few
>>> minutes to scratch an idea of something that could be of interest to
>>> others, such as "how we deal with logs at XXX", "how we keep our configs
>>> in sync", "how we built our own CDN", "12 things to take care of for
>>> scalability", "how we perform A/B testing", etc. Also if you contribute
>>> to other OSS projects and see certain things done well there that you
>>> think haproxy could benefit from, that could bring interesting discussions
>>> as well! Anyway I'll ping again and more aggressively next week!
>> 
>> 
>> Well, I guess now I *have* to submit that talk idea about how we built a
>> global PoP network, saved 80-90% auth overhead per request, and saved money
>> at the same time... :-)
> 
> Oh yes definitely!
> 
>> If we just use the form on the website, does that suffice or should we also
>> submit to submissions@?
> 
> I don't know :-)  I simply sent an email to submissi...@haproxy.com 
>  with
> my talk proposal (how to most efficiently contribute), I guess it will work
> fine.
> 
> Thanks,
> Willy

Re: [ANNOUNCE] haproxy-2.0-dev6 + Conf reminder

[Willy Tarreau]

On Fri, Jun 07, 2019 at 10:03:59AM +0200, Luke Seelenbinder wrote:
> > On another point, I was told that very few proposals for talks at the
> > HAProxyConf were sent to date. That doesn't surprize me and I haven't
> > even sent mine yet either :-)  Please keep in mind that the deadline for
> > proposals submissions is in two weeks and that time flies. So instead of
> > wasting your time reading my lengthy boring e-mails, please take a few
> > minutes to scratch an idea of something that could be of interest to
> > others, such as "how we deal with logs at XXX", "how we keep our configs
> > in sync", "how we built our own CDN", "12 things to take care of for
> > scalability", "how we perform A/B testing", etc. Also if you contribute
> > to other OSS projects and see certain things done well there that you
> > think haproxy could benefit from, that could bring interesting discussions
> > as well! Anyway I'll ping again and more aggressively next week!
> 
> 
> Well, I guess now I *have* to submit that talk idea about how we built a
> global PoP network, saved 80-90% auth overhead per request, and saved money
> at the same time... :-)

Oh yes definitely!

> If we just use the form on the website, does that suffice or should we also
> submit to submissions@?

I don't know :-)  I simply sent an email to submissi...@haproxy.com with
my talk proposal (how to most efficiently contribute), I guess it will work
fine.

Thanks,
Willy

Re: [ANNOUNCE] haproxy-2.0-dev6 + Conf reminder

[Luke Seelenbinder]

> On another point, I was told that very few proposals for talks at the
> HAProxyConf were sent to date. That doesn't surprize me and I haven't
> even sent mine yet either :-)  Please keep in mind that the deadline for
> proposals submissions is in two weeks and that time flies. So instead of
> wasting your time reading my lengthy boring e-mails, please take a few
> minutes to scratch an idea of something that could be of interest to
> others, such as "how we deal with logs at XXX", "how we keep our configs
> in sync", "how we built our own CDN", "12 things to take care of for
> scalability", "how we perform A/B testing", etc. Also if you contribute
> to other OSS projects and see certain things done well there that you
> think haproxy could benefit from, that could bring interesting discussions
> as well! Anyway I'll ping again and more aggressively next week!


Well, I guess now I *have* to submit that talk idea about how we built a global 
PoP network, saved 80-90% auth overhead per request, and saved money at the 
same time… :-)

If we just use the form on the website, does that suffice or should we also 
submit to submissions@?

Best,
Luke

—
Luke Seelenbinder
Stadia Maps | Founder
stadiamaps.com

Re: [ANNOUNCE] haproxy-2.0-dev6 + Conf reminder

[Willy Tarreau]

On Fri, Jun 07, 2019 at 07:35:38AM +0200, Willy Tarreau wrote:
> On another point, I was told that very few proposals for talks at the
> HAProxyConf were sent to date. That doesn't surprize me and I haven't
> even sent mine yet either :-)  Please keep in mind that the deadline for
> proposals submissions is in two weeks and that time flies. So instead of
> wasting your time reading my lengthy boring e-mails, please take a few
> minutes to scratch an idea of something that could be of interest to
> others, such as "how we deal with logs at XXX", "how we keep our configs
> in sync", "how we built our own CDN", "12 things to take care of for
> scalability", "how we perform A/B testing", etc. Also if you contribute
> to other OSS projects and see certain things done well there that you
> think haproxy could benefit from, that could bring interesting discussions
> as well! Anyway I'll ping again and more aggressively next week!

While I was looking for the address to submit my talk proposal, I realized
I hadn't posted it again, so here it comes so that it cannot serve as an
excuse for not proposing anything :-)

  submiss...@haproxy.com

Willy

[ANNOUNCE] haproxy-2.0-dev6 + Conf reminder

[Willy Tarreau]

Hi,

HAProxy 2.0-dev6 was released on 2019/06/07. It added 73 new commits
after version 2.0-dev5.

The good thing is that all the pending stuff I was aware of was finally
merged in time. The bad thing is that we had to address yet another
problem with connection setup expiration causing loops in process_stream()
and that the fix required deep changes that I didn't feel comfortable with
at this stage of the cycle. But better fix them before the release than
papering over the issue afterwards. I'm now certain that 1.9 is affected
as well, and just thinking that at some point I'll have to backport this
changes to fix 1.9 isn't the first thing that makes me want to wake up
every morning. But while I'm still not certain that I perfectly addressed
100% of the remaining issues in this category, I'm at least certain that
it's way better now than it was and it will be easier in the future to
address any possible remaining issue. So we'll observe on 2.0 and decide
to backport on 1.9 once we're more confident.

I'm quite happy with the latest merges because they will be useful over
the long term for this release, present very low risks and will really
help with the maintenance, so special thanks to those who made efforts
to send their code in time! Among the latest changes I can enumerate in
no particular order :

  - Fix of the HTX encoding of HTTP trailers : it used to be a plain
copy of the HTTP/1 representation which is good neither for H2 nor
for captures or later processing. Now they are encoded just like
headers, which means that it'll be easier to perform minor updates
on them later if needed (e.g. Patrick asked for a way to capture
them, which I still don't know how to do but will surely be easier
now).

  - The new stick-table data type "server_name" was added. It works just
like server_id except that what is exchanged over the wire is the
server's name in addition of its ID (and MagicFred managed to keep it
backwards compatible with older versions, so no protocol upgrade is
needed). While IDs are useful for globally maintained configurations,
they don't work well in highly dynamic environments where servers are
inserted and removed in any order. Now nothing really requires to use
a specific server ID anymore, which confirms our move towards making
the config easier to integrate with modern tools. By the way we've
seen how we could further extend the current mechanism to stick on a
server's address later if needed (which could be nice with DNS).

  - The server-side SOCKS4 layer was merged. The code was clean and
represents very little risk. Theorically we ought to have postponed
this to 2.1 since it was not initially planned but I'm not in favor
of bureaucratic rules so I prefered to take it anyway :-)

  - Completion of the stacking of the connection layers : for a long
time (since 1.5) we used to have two polling modes for connections,
depending on handshakes or data. This was extremely tricky and
regularly caused some issues (such as the recent SOCKS4 issues).
The split of these in layers was started just after 1.9 and the
remaining parts (replacement of flags with subscriptions) was just
completed, making the whole stack work the same from bottom to top,
and removing lots of exceptions everywhere in the code. This will
help us get rid of the old model to improve long term maintenance
and reduce the number of bugs.

  - A few sample fetch functions for SSL were added. They're supposed
to help debugging from traffic captures. I don't know if they're
needed in older versions (nor if they are compatible), but I'm not
against backporting low-risk stuff that helps with debugging like
this.

  - The server connection pool default purge delay was extended from 1
to 5 seconds : I noticed during some testing that when you have many
servers (hence few connections per server), starting to kill idle
connections after just one second provided a very low reuse rate, so
let's raise this one a little bit. This is just default tuning anyway.

The rest is just cleanups, build fixes and bug fixes.

Overall the bug fixing rate is going down significantly and their severity
as well, so I'm reasonably confident in what we have. I recently said that
I expect to release next week, but we've slipped a little bit due to the
painful issue described above. Well, we'll see how things go.

I'm aware of a few minor things that need to be addressed but that are not
even show-stoppers for the release :
  - tcp-request inspect delay is currently not respected in HTX mode,
but Christopher knows why, this is just a silly thing that needs to
be addressed, hopefully before the release.

  - the state-file still compares server IDs and names and tries to do its
best. It would be nice if it now favored the name over the ID (or even
ignored the ID). Baptiste told me he'll have a look