Re: [ANNOUNCE] haproxy-1.8.13
Willy, Am 31.07.2018 um 20:32 schrieb Willy Tarreau: > That's where I disagree, it's exactly the same argument causing TLS to > appear on every web site even when not necessary, making people believe > they are safe while they are not. Right now you don't have this PGP > signature so you are careful about what you retrieve. And that's the > reason why you're talking about it by the way, because verifying all > this is painful on your side. But if I start to claim "look, no need > to double-check anymore, trust me, it's safe", you won't run this > extra safety check once in a while. But the process involved in placing > this signature may not be safer than the one involved in the checksum. > > With this said, I'll take a look at Bertrand's proposal, which I think > does satisfy my needs. To nitpick this still would require you to trust the binaries (e.g. tar) on the haproxy.org machine :-) Anyway: I am disgressing here and will patiently await whether or not there will be PGP signatures in the future. Best regards Tim Düsterhus
Re: [ANNOUNCE] haproxy-1.8.13
On Tue, Jul 31, 2018 at 07:42:41PM +0200, Tim Düsterhus wrote: > Am 30.07.2018 um 20:55 schrieb Willy Tarreau: > > I know and I've already thought about it. But I personally refuse to store > > my PGP key on any exposed machine. Right now in order to tag, I have to > > SSH into an isolated machine, run "git pull --tags", create-release, and > > "git push --tags". Then I upload the release. > > In addition to what Vincent and Bertrand suggest I'd like to note that a > dedicated "haproxy Release Signing Key", even if stored on an exposed > machine, would be strictly better than just checksums, which could be > modified by anyone with access to the haproxy.org server. That's where I disagree, it's exactly the same argument causing TLS to appear on every web site even when not necessary, making people believe they are safe while they are not. Right now you don't have this PGP signature so you are careful about what you retrieve. And that's the reason why you're talking about it by the way, because verifying all this is painful on your side. But if I start to claim "look, no need to double-check anymore, trust me, it's safe", you won't run this extra safety check once in a while. But the process involved in placing this signature may not be safer than the one involved in the checksum. With this said, I'll take a look at Bertrand's proposal, which I think does satisfy my needs. > Also I know nothing about the release process, but: Is the machine > signing the tags not used to upload the release Tarballs to haproxy.org? It depends who does it. Speaking for myself, since my PGP key is not on the machine, I release using create-release (changelog+commit+signed tag) on the machine where I have the PGP key, then I push to the public repo, then I connect to the haproxy.org machine to publish the release from the latest tag using the publish-release utility you recently patched, and perform a few extra actions there to automatically update the home page and the known bugs page. Then I run announce-release from any machine, which prepares a horrible automated text that will serve as a basis for the announce. > I think it's strange that the parts of the release process are > distributed onto several machines (one to create the tag, one to create > the Tarball). No it's not uncommon at all, especially with git since signed tags can be done anywhere, especially at places where you don't want to upload large tarballs when you in fact only need to upload a tag. Imagine if I had had to upload full linux kernels when doing stable releases, it would have taken many hours just to upload the tarballs! Cheers, Willy
Re: [ANNOUNCE] haproxy-1.8.13
Hi Bertrand, On Tue, Jul 31, 2018 at 06:26:11PM +0100, Bertrand Jacquin wrote: > I know old farts don't change, but for the two cents, newer version of > OpenSSH (>= 6.7) and GnuPG (>=2.1.1) allow you to forward GnuPG agent over > SSH with reduce capacity to reduce the attack surface you are mentioning. > More details are available on https://wiki.gnupg.org/AgentForwarding Thanks for the info, I can possibly have a look at this. It could be a reasonable option indeed, which limits the exposure of the agent in time. Cheers, Willy
Re: [ANNOUNCE] haproxy-1.8.13
Willy, Am 30.07.2018 um 20:55 schrieb Willy Tarreau: > I know and I've already thought about it. But I personally refuse to store > my PGP key on any exposed machine. Right now in order to tag, I have to > SSH into an isolated machine, run "git pull --tags", create-release, and > "git push --tags". Then I upload the release. In addition to what Vincent and Bertrand suggest I'd like to note that a dedicated "haproxy Release Signing Key", even if stored on an exposed machine, would be strictly better than just checksums, which could be modified by anyone with access to the haproxy.org server. This signing key could be signed by your personal PGP key and easily be revoked in case it ever gets compromised. Also I know nothing about the release process, but: Is the machine signing the tags not used to upload the release Tarballs to haproxy.org? I think it's strange that the parts of the release process are distributed onto several machines (one to create the tag, one to create the Tarball). Best regards Tim Düsterhus
Re: [ANNOUNCE] haproxy-1.8.13
On 31/07/2018 18:26, Bertrand Jacquin wrote: Hi Willy, On 30/07/2018 19:55, Willy Tarreau wrote: On Mon, Jul 30, 2018 at 07:41:33PM +0200, Tim Düsterhus wrote: Willy, Am 30.07.2018 um 18:05 schrieb Willy Tarreau: > A small update happened to the download directory, the sha256 of the > tar.gz files are now present in addition to the (quite old) md5 ones. > We may start to think about phasing md5 signatures out, for example > after 1.9 is released. I'd even like to see PGP signatures, like you already do for the git tags (but not the Tarballs). But this is a greater change than just updating the checksums :-) I know and I've already thought about it. But I personally refuse to store my PGP key on any exposed machine. Right now in order to tag, I have to SSH into an isolated machine, run "git pull --tags", create-release, and "git push --tags". Then I upload the release. What I don't like with PGP on an exposed machine is that it reduces the size of your 4096-bit key to the size of your passphrase (which most often contains much less than the ~700 characters it would need to be as large), and also increases your ability to get fooled into entering it. Some would call me paranoid, but I don't think I am, I'm just trying to keep a balanced level of security, knowing that the global one is not better than the weakest point. If I wanted to sign the images, it would require to find a different release method and would significantly complicate the procedure. I know old farts don't change, but for the two cents, newer version of OpenSSH (>= 6.7) and GnuPG (>=2.1.1) allow you to forward GnuPG agent over SSH with reduce capacity to reduce the attack surface you are mentioning. More details are available on https://wiki.gnupg.org/AgentForwarding Also, old farts press the send button too quickly. The benefit of forwarding the gpg agent is that you don't need to copy your private key to any remote machine, the gpg agent running on the machine forwarding it will perform all the crypto operations. With a ssh config alias, you could enable agent forwarding only when you want to create a release. Mixed with a smartcard, no computer at all would be able to access private material. Cheers -- Bertrand
Re: [ANNOUNCE] haproxy-1.8.13
Hi Willy, On 30/07/2018 19:55, Willy Tarreau wrote: On Mon, Jul 30, 2018 at 07:41:33PM +0200, Tim Düsterhus wrote: Willy, Am 30.07.2018 um 18:05 schrieb Willy Tarreau: > A small update happened to the download directory, the sha256 of the > tar.gz files are now present in addition to the (quite old) md5 ones. > We may start to think about phasing md5 signatures out, for example > after 1.9 is released. I'd even like to see PGP signatures, like you already do for the git tags (but not the Tarballs). But this is a greater change than just updating the checksums :-) I know and I've already thought about it. But I personally refuse to store my PGP key on any exposed machine. Right now in order to tag, I have to SSH into an isolated machine, run "git pull --tags", create-release, and "git push --tags". Then I upload the release. What I don't like with PGP on an exposed machine is that it reduces the size of your 4096-bit key to the size of your passphrase (which most often contains much less than the ~700 characters it would need to be as large), and also increases your ability to get fooled into entering it. Some would call me paranoid, but I don't think I am, I'm just trying to keep a balanced level of security, knowing that the global one is not better than the weakest point. If I wanted to sign the images, it would require to find a different release method and would significantly complicate the procedure. I know old farts don't change, but for the two cents, newer version of OpenSSH (>= 6.7) and GnuPG (>=2.1.1) allow you to forward GnuPG agent over SSH with reduce capacity to reduce the attack surface you are mentioning. More details are available on https://wiki.gnupg.org/AgentForwarding Cheers -- Bertrand
Re: [ANNOUNCE] haproxy-1.8.13
Hi Vincent, On Mon, Jul 30, 2018 at 11:16:39PM +0200, Vincent Bernat wrote: > ? 30 juillet 2018 20:55 +0200, Willy Tarreau : > > > What I don't like with PGP on an exposed machine is that it reduces the > > size of your 4096-bit key to the size of your passphrase (which most > > often contains much less than the ~700 characters it would need to be > > as large), and also increases your ability to get fooled into entering > > it. Some would call me paranoid, but I don't think I am, I'm just trying > > to keep a balanced level of security, knowing that the global one is not > > better than the weakest point. > > Attacks on asymmetric ciphers do not rely on bruteforce: you don't have > to explore the whole keyspace to guess the private key. You can use > algorithms like the general number field sieve. A 4096-bit RSA keypair > would be roughly equivalent to a symmetric algorithm using a 160-bit key > (unless we find better algorithms to break RSA). I thought RSA4096 was equivalent to more than this, I'm disappointed :-) > A 32-character > passphrase would be enough to protect the private key. Moreover, if you > use a weaker passphrase, you have not lost yet as the string to key > function used to turn the passphrase into an AES key is slow. I don't > know where the limit is, but the idea is that with a shorter passphrase, > the attacker may still have a better time finding the AES key instead of > the passphrase. I see, the same principle as system passwords using many rounds to slow down brute force attacks. With this said, when you see the amount of power that some ASICs, FPGAs and GPUs have developed over the years due to the mining activities, often counting in gigahashes/s, I suspect you'll need many rounds to be safe :-/ > But if someone can steal your encrypted key from your machine, they may > also be able to steal the unencrypted one through various means. So, you > may still be right about being paranoid. :) Yes, that's still the point. After all, when you have access to a user- owned file, you also have access to this user's processes. It's not very complicated to run "while ! strace -o foo.log -p $(pgrep gpg); do sleep 0.1;done", it remains very discrete and will easily reveal the passphrase. Thanks for the detailed explanation! Willy
Re: [ANNOUNCE] haproxy-1.8.13
❦ 30 juillet 2018 20:55 +0200, Willy Tarreau : > What I don't like with PGP on an exposed machine is that it reduces the > size of your 4096-bit key to the size of your passphrase (which most > often contains much less than the ~700 characters it would need to be > as large), and also increases your ability to get fooled into entering > it. Some would call me paranoid, but I don't think I am, I'm just trying > to keep a balanced level of security, knowing that the global one is not > better than the weakest point. Attacks on asymmetric ciphers do not rely on bruteforce: you don't have to explore the whole keyspace to guess the private key. You can use algorithms like the general number field sieve. A 4096-bit RSA keypair would be roughly equivalent to a symmetric algorithm using a 160-bit key (unless we find better algorithms to break RSA). A 32-character passphrase would be enough to protect the private key. Moreover, if you use a weaker passphrase, you have not lost yet as the string to key function used to turn the passphrase into an AES key is slow. I don't know where the limit is, but the idea is that with a shorter passphrase, the attacker may still have a better time finding the AES key instead of the passphrase. But if someone can steal your encrypted key from your machine, they may also be able to steal the unencrypted one through various means. So, you may still be right about being paranoid. :) -- The man who sets out to carry a cat by its tail learns something that will always be useful and which never will grow dim or doubtful. -- Mark Twain
Re: [ANNOUNCE] haproxy-1.8.13
On Mon, Jul 30, 2018 at 07:41:33PM +0200, Tim Düsterhus wrote: > Willy, > > Am 30.07.2018 um 18:05 schrieb Willy Tarreau: > > A small update happened to the download directory, the sha256 of the > > tar.gz files are now present in addition to the (quite old) md5 ones. > > We may start to think about phasing md5 signatures out, for example > > after 1.9 is released. > > I'd even like to see PGP signatures, like you already do for the git > tags (but not the Tarballs). But this is a greater change than just > updating the checksums :-) I know and I've already thought about it. But I personally refuse to store my PGP key on any exposed machine. Right now in order to tag, I have to SSH into an isolated machine, run "git pull --tags", create-release, and "git push --tags". Then I upload the release. What I don't like with PGP on an exposed machine is that it reduces the size of your 4096-bit key to the size of your passphrase (which most often contains much less than the ~700 characters it would need to be as large), and also increases your ability to get fooled into entering it. Some would call me paranoid, but I don't think I am, I'm just trying to keep a balanced level of security, knowing that the global one is not better than the weakest point. If I wanted to sign the images, it would require to find a different release method and would significantly complicate the procedure. Willy
Re: [ANNOUNCE] haproxy-1.8.13
Willy, Am 30.07.2018 um 18:05 schrieb Willy Tarreau: > A small update happened to the download directory, the sha256 of the > tar.gz files are now present in addition to the (quite old) md5 ones. > We may start to think about phasing md5 signatures out, for example > after 1.9 is released. I'd even like to see PGP signatures, like you already do for the git tags (but not the Tarballs). But this is a greater change than just updating the checksums :-) Best regards Tim Düsterhus
Re: [ANNOUNCE] haproxy-1.8.13
On 30/07/2018 18:05, Willy Tarreau wrote: Hi, HAProxy 1.8.13 was released on 2018/07/30. It added 28 new commits after version 1.8.12. Nothing critical this time, however we finally got rid of the annoying CLOSE_WAIT on H2 thanks to the continued help from Milan Petruzelka, Janusz Dziemidowicz and Olivier Doucet. Just for this it was worth emitting a release. During all these tests we also met a case where sending a POST to the stats applet over a slow link using H2 could sometimes result in haproxy busy waiting for data, causing 100% CPU being seen. It was fixed, along with another bug affecting applets like stats, possibly causing occasional CPU spikes. While developing on 1.9 we found a few interesting corner cases with threads, one of which causes performance to significantly drop when reaching a server maxconn *if* there are more threads than available CPUs. It turned out to be caused by the synchronization point not leaving enough CPU to sleeping threads to be scheduled and join. You should never ever use less threads than CPUs, but config errors definitely happen and we'd rather limit their impact. Speaking about config errors, another case existed where a "process" directive on a "bind" line could reference non-existing threads. If only non-existing threads were referenced, it didn't trigger an error and would silently start, but with nobody to accept the traffic. It easily happens when reducing the number of threads in a config. This was addressed similarly to the process case, where the threads are automatically remapped and a warning is emitted in this case. An issue was addressed with the proxy protocol header sent to servers. If a "http-request set-src" directive is used, it is possible to end up with a mix of IPv4 and IPv6, which cannot be transported by the protocol (since it makes no sense from a network perspective). Till now a server would only receive "PROXY UNKNOWN" and would not even be able to get the client's address. Tim Duesterhus addressed this by converting the IPv4 address to IPv6 if exactly one of the addresses is IPv6. It is the only way not to lose information Christopher addressed a rare issue which could trigger during soft reloads with threads enabled : if a thread quits at the exact moment a thread sync is requested, the remaining threads could wait for it forever. Vincent Bernat updated the systemd unit file so that when quitting, if the master reports 143 (SIGTERM+128) as the exit status due to the fact that it reports the last killed worker's status, systemd doesn't consider this as a failure. The remaining changes are pretty minor. Some H2 debugging code developed to fix the CLOSE_WAIT issues was backported in orther to simplify the retrieval of internal states when such issue shappen. A small update happened to the download directory, the sha256 of the tar.gz files are now present in addition to the (quite old) md5 ones. We may start to think about phasing md5 signatures out, for example after 1.9 is released. As usual, it's worth updating if you're on 1.8, especially if you're using H2 and/or threads. If you think you've found a bug that is not addressed in the changelog below, please update and try again before reporting it. There are so many possible side effects from H2 issues and thread issues that it is possible that your issue is a different manifestation of one of these. Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Sources : http://www.haproxy.org/download/1.8/src/ Git repository : http://git.haproxy.org/git/haproxy-1.8.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-1.8.git Changelog: http://www.haproxy.org/download/1.8/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ As always the new Version is also on the docker hub. https://hub.docker.com/r/me2digital/haproxy18/ Willy Regards Aleks --- Complete changelog : Christopher Faulet (4): BUG/MINOR: http: Set brackets for the unlikely macro at the right place MINOR: debug: Add check for CO_FL_WILL_UPDATE MINOR: debug: Add checks for conn_stream flags BUG/MEDIUM: threads: Fix the exit condition of the thread barrier Olivier Houchard (2): BUG/MINOR: servers: Don't make "server" in a frontend fatal. BUG/MINOR: threads: Handle nbthread == MAX_THREADS. Tim Duesterhus (2): BUILD: Generate sha256 checksums in publish-release MEDIUM: proxy_protocol: Convert IPs to v6 when protocols are mixed Vincent Bernat (1): MINOR: systemd: consider exit status 143 as successful Willy Tarreau (19): BUG/MINOR: ssl: properly ref-count the tls_keys entries MINOR: mux: add a "show_fd" function to dump debugging information for "show fd" MINOR: h2: implement a basic "show_fd" function BUG/MINOR: h2: remove accidental debug code introduced with show_fd function MINOR: h2: keep a count of the
[ANNOUNCE] haproxy-1.8.13
Hi, HAProxy 1.8.13 was released on 2018/07/30. It added 28 new commits after version 1.8.12. Nothing critical this time, however we finally got rid of the annoying CLOSE_WAIT on H2 thanks to the continued help from Milan Petruzelka, Janusz Dziemidowicz and Olivier Doucet. Just for this it was worth emitting a release. During all these tests we also met a case where sending a POST to the stats applet over a slow link using H2 could sometimes result in haproxy busy waiting for data, causing 100% CPU being seen. It was fixed, along with another bug affecting applets like stats, possibly causing occasional CPU spikes. While developing on 1.9 we found a few interesting corner cases with threads, one of which causes performance to significantly drop when reaching a server maxconn *if* there are more threads than available CPUs. It turned out to be caused by the synchronization point not leaving enough CPU to sleeping threads to be scheduled and join. You should never ever use less threads than CPUs, but config errors definitely happen and we'd rather limit their impact. Speaking about config errors, another case existed where a "process" directive on a "bind" line could reference non-existing threads. If only non-existing threads were referenced, it didn't trigger an error and would silently start, but with nobody to accept the traffic. It easily happens when reducing the number of threads in a config. This was addressed similarly to the process case, where the threads are automatically remapped and a warning is emitted in this case. An issue was addressed with the proxy protocol header sent to servers. If a "http-request set-src" directive is used, it is possible to end up with a mix of IPv4 and IPv6, which cannot be transported by the protocol (since it makes no sense from a network perspective). Till now a server would only receive "PROXY UNKNOWN" and would not even be able to get the client's address. Tim Duesterhus addressed this by converting the IPv4 address to IPv6 if exactly one of the addresses is IPv6. It is the only way not to lose information Christopher addressed a rare issue which could trigger during soft reloads with threads enabled : if a thread quits at the exact moment a thread sync is requested, the remaining threads could wait for it forever. Vincent Bernat updated the systemd unit file so that when quitting, if the master reports 143 (SIGTERM+128) as the exit status due to the fact that it reports the last killed worker's status, systemd doesn't consider this as a failure. The remaining changes are pretty minor. Some H2 debugging code developed to fix the CLOSE_WAIT issues was backported in orther to simplify the retrieval of internal states when such issue shappen. A small update happened to the download directory, the sha256 of the tar.gz files are now present in addition to the (quite old) md5 ones. We may start to think about phasing md5 signatures out, for example after 1.9 is released. As usual, it's worth updating if you're on 1.8, especially if you're using H2 and/or threads. If you think you've found a bug that is not addressed in the changelog below, please update and try again before reporting it. There are so many possible side effects from H2 issues and thread issues that it is possible that your issue is a different manifestation of one of these. Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Sources : http://www.haproxy.org/download/1.8/src/ Git repository : http://git.haproxy.org/git/haproxy-1.8.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-1.8.git Changelog: http://www.haproxy.org/download/1.8/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ Willy --- Complete changelog : Christopher Faulet (4): BUG/MINOR: http: Set brackets for the unlikely macro at the right place MINOR: debug: Add check for CO_FL_WILL_UPDATE MINOR: debug: Add checks for conn_stream flags BUG/MEDIUM: threads: Fix the exit condition of the thread barrier Olivier Houchard (2): BUG/MINOR: servers: Don't make "server" in a frontend fatal. BUG/MINOR: threads: Handle nbthread == MAX_THREADS. Tim Duesterhus (2): BUILD: Generate sha256 checksums in publish-release MEDIUM: proxy_protocol: Convert IPs to v6 when protocols are mixed Vincent Bernat (1): MINOR: systemd: consider exit status 143 as successful Willy Tarreau (19): BUG/MINOR: ssl: properly ref-count the tls_keys entries MINOR: mux: add a "show_fd" function to dump debugging information for "show fd" MINOR: h2: implement a basic "show_fd" function BUG/MINOR: h2: remove accidental debug code introduced with show_fd function MINOR: h2: keep a count of the number of conn_streams attached to the mux MINOR: h2: add the mux and demux buffer lengths on "show fd" BUG/MEDIUM: h2: don't accept new
