Re: Issue with TCP splicing
private keys private key defiantly need revoked i attached key to show how easy it is to xtract from coredump you can download certificate https://crt.sh/?d=511527676 and verify signature of message.txt using openssl smime -verify -in message.txt.p7s -content message.txt -inform PEM -noverify 511527676.crt Private-Key: (2048 bit) modulus: 00:b8:94:de:cc:4f:9a:a5:2b:d5:56:4f:62:3c:c1: 78:75:e4:ed:b8:f5:1d:2c:d3:27:2a:01:de:39:72: 4d:ef:54:db:d7:a2:e2:e3:ed:a5:6c:36:f4:fc:d0: 1f:3e:07:20:e6:b7:e3:4b:43:70:63:99:d1:df:58: bb:1e:c1:b4:61:81:48:38:da:00:8f:0f:62:f8:d3: 86:70:bc:3b:d4:0d:ad:ce:b6:53:a3:fe:0e:fb:d6: d0:bf:13:e9:b7:a3:7c:2c:10:06:41:6b:15:aa:81: 41:89:23:7d:32:27:a7:74:50:94:8e:15:67:0b:5c: ad:51:f1:58:24:d8:88:02:62:32:e9:de:a7:5b:8a: cc:ff:fc:d1:9b:f5:6e:17:2f:bc:0f:a6:d5:9f:26: f4:a8:f7:48:9a:37:3f:22:f1:8f:77:70:38:28:96: d7:6f:af:2d:de:74:32:2c:e5:21:6b:df:0b:41:b8: f2:d6:5c:91:17:70:56:ad:6d:71:e4:b1:a5:2a:65: ca:51:f4:ec:b6:fb:8b:d1:f3:bf:cf:19:83:d5:d9: 61:03:c1:87:7d:8d:27:4e:f4:d6:e4:5d:4f:56:cc: 02:c3:b1:73:63:24:38:2f:e1:5c:19:94:c4:6d:40: 82:43:ef:6d:15:98:04:73:47:d7:c2:c9:11:46:e4: 3c:0f publicExponent: 65537 (0x10001) privateExponent: 18:94:fa:f7:0a:c2:f5:ac:58:c5:1d:dd:5f:6a:04: b8:ee:bc:1a:1d:ca:bc:e5:82:19:be:15:f2:60:9e: b0:79:04:ae:3b:2b:2c:5f:c1:e0:1f:91:90:f9:c6: af:64:13:a5:a6:67:c6:e6:3c:59:87:6a:c3:eb:f5: 3f:ab:5c:72:7f:dd:36:75:12:0d:fb:66:9a:ec:d0: c2:c2:ce:d4:f6:dd:66:e2:31:51:6d:cc:61:0d:c2: cf:2f:bf:b8:8d:35:44:48:fe:0c:48:4e:a2:5e:84: 73:d7:1e:1d:47:da:ad:4a:ed:fd:de:2b:d2:ff:8c: b5:95:06:c0:21:76:3b:9a:ce:06:86:88:4f:b2:6a: 2f:e0:84:79:d0:e4:cd:6c:8a:cf:33:3e:fd:43:da: e3:63:c0:d6:11:c0:12:ec:2f:85:7d:f8:23:67:b7: 6d:5d:c6:d6:2e:99:28:7d:2b:40:6e:4f:f5:d5:55: b9:01:97:4b:d4:08:14:2d:71:19:9e:e4:0b:f3:0f: 6e:a2:4a:9f:fe:fb:34:37:d4:b7:e3:ce:45:c9:c4: 41:07:69:45:71:37:30:c7:fc:3b:1e:49:bd:7a:c4: f3:02:82:55:6c:a5:de:47:62:f1:a8:09:16:61:05: 8c:df:3f:62:6c:fb:5a:28:36:2f:70:f0:ff:28:dd: 81 prime1: 00:c0:3c:af:12:53:99:c5:0a:f7:32:7e:f7:74:5d: d6:67:a8:f2:6a:03:f4:97:28:e6:e8:ab:e6:54:35: b9:d7:e9:2d:11:df:76:01:0f:6a:af:91:9d:8a:b1: 79:ae:45:8e:b9:23:a0:f3:35:2f:65:a2:8c:d2:5e: 8f:ba:53:86:53:96:b1:5d:10:e1:57:90:31:47:d5: e9:b3:62:17:72:c8:23:ab:d7:ea:c4:7c:67:32:63: 0f:ef:f0:d5:30:04:7d:09:e5:da:5d:4d:d6:32:3e: 9c:f1:4c:95:f9:f9:c8:63:15:5d:ed:bf:fa:a5:19: 41:8b:fd:39:61:a5:5e:e3:99 prime2: 00:f5:ce:23:2e:12:a7:c4:13:ae:70:95:ad:88:34: 43:bc:3d:76:c2:e1:45:d6:21:8a:80:7e:50:44:a8: cf:76:46:4c:9a:dd:6d:f4:06:f2:f2:91:aa:53:45: 43:eb:5c:00:87:7e:d9:02:42:66:2e:1d:08:79:fa: 3b:2f:bb:e0:10:bb:26:d5:db:6c:7a:19:9a:4d:be: f1:26:02:b2:93:3c:67:46:92:09:9c:d9:6b:82:d3: 0b:b4:e1:63:d8:1c:e9:4c:77:50:b6:1d:50:09:d8: 79:a5:6e:94:6a:4a:d0:3b:de:e1:db:44:5b:80:76: 4f:f7:13:05:3b:3e:35:e5:e7 exponent1: 63:39:ef:94:22:1a:e9:1e:73:e2:58:af:1a:1d:a5: a1:f4:0e:cc:b2:25:fa:30:5e:a0:12:ba:dd:14:ae: 4c:c8:4b:3f:42:7d:02:a7:16:86:71:3f:44:6b:bf: 47:39:18:26:70:41:8f:c8:10:23:01:f8:76:4d:e1: 1a:68:2a:99:d2:da:d2:12:f8:7d:de:2b:d1:cc:94: c8:c7:05:1b:76:3b:13:64:6c:05:e7:c0:cc:bd:5d: 68:98:83:32:39:de:e0:d1:08:19:c9:27:9a:df:be: da:be:91:5b:6a:97:08:ad:ea:c1:e1:aa:5a:b5:e2: a3:83:9d:ae:cd:51:61:61 exponent2: 00:9a:d4:72:a2:75:cb:c9:1d:60:96:b8:21:6b:97: 08:47:8d:2b:be:8b:69:92:fc:e3:a2:16:6e:77:21: 22:34:ed:09:19:cf:7a:8f:e8:c4:a5:78:8d:a2:10: 12:3d:31:61:7f:f7:ad:b7:d7:9d:47:54:b0:5f:2c: f8:95:13:b1:8a:b8:68:38:f3:12:fc:42:1e:48:f4: 8a:2f:98:29:65:c6:f9:82:a1:40:7e:d5:10:fc:81: f5:70:c5:3c:40:07:ce:08:85:6b:88:9b:24:2c:5f: 78:18:75:73:f5:14:14:e0:71:7f:30:bf:79:27:8c: de:c7:d1:ea:4c:ab:de:05:67 coefficient: 79:98:31:aa:49:d9:02:cb:2b:c5:f6:a3:33:32:ca: 97:a1:12:28:6b:e5:9a:48:6e:47:bf:01:59:7c:e6: a1:78:8e:dc:cf:f4:69:b7:9b:f9:f3:5b:84:98:cd: 2f:1f:71:7b:e8:10:e0:55:f4:c0:f1:59:5c:72:05: aa:af:96:56:68:53:e1:9e:25:84:f9:fc:a9:2b:29: 61:60:42:55:a9:05:3f:0c:db:0b:f1:a6:62:cb:69: e1:c3:c4:35:ed:fc:94:4e:24:16:f4:66:7a:03:5e: e0:8e:af:50:21:63:cd:f4:ae:fe:9e:da:07:0b:e9: 8c:7c:7b:fa:c3:60:a5:bacertificate is compromised -BEGIN PKCS7- MIIIkAYJKoZIhvcNAQcCoIIIgTCCCH0CAQExDzANBglghkgBZQMEAgEFADALBgkq hkiG9w0BBwGgggV/MIIFezCCBGOgAwIBAgIJALUFEZ7ciuLlMA0GCSqGSIb3DQEB CwUAMIG0MQswCQYDVQQGEwJVUzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMK U2NvdHRzZGFsZTEaMBgGA1UEChMRR29EYWRkeS5jb20sIEluYy4xLTArBgNVBAsT JGh0dHA6Ly9jZXJ0cy5nb2RhZGR5LmNvbS9yZXBvc2l0b3J5LzEzMDEGA1UEAxMq R28gRGFkZHkgU2VjdXJlIENlcnRpZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTE4 MDYwNzEyMjQ0OFoXDTIwMDYyODEzMDAzMVowTDEhMB8GA1UECxMYRG9tYWluIENv bnRyb2wgVmFsaWRhdGVkMScwJQYDVQQDEx5wYWNrZXRmZW5jZS5jc2NoaWMtY2hv Y3MucWMuY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC4lN7MT5ql
Re: Issue with TCP splicing
Hello Julien, On Thu, 23 Aug 2018 at 20:49, Julien Semaan wrote: > > Hi Olivier, > > Sorry for the delay, obtaining the core dump from a production environment > was a bit tricky. > > So, I have attached the core dump to this email. I hope this will help you > identify the issue. The executable is also needed to analyze the coredump, please share it as well; also you probably want to be more careful sending around those core-dumps. It can contain all kinds of private data, like private keys, and unencrypted HTTP data, etc. By sending it to the list, you shared it with everyone, usually you'd just unicast it to whoever is working with you on this crash. Regards, Lukas
Re: Issue with TCP splicing
On Wed, Jul 25, 2018 at 09:02:24AM -0400, Julien Semaan wrote: > Hi Olivier, > > Thanks for the time you're taking to check the issue. > > I'll get an environment back with TCP splicing enabled and I'll run it in > GDB and provide you a core dump > That would be great, thank you ! Olivier
Re: Issue with TCP splicing
Hi Olivier, Thanks for the time you're taking to check the issue. I'll get an environment back with TCP splicing enabled and I'll run it in GDB and provide you a core dump Best Regards, -- Julien Semaan jsem...@inverse.ca :: +1 (866) 353-6153 *155 ::www.inverse.ca Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org) On 2018-07-25 08:59 AM, Olivier Houchard wrote: Hi Julien, On Tue, Jul 24, 2018 at 01:29:49PM -0400, Julien Semaan wrote: Sorry, that was a "can" that really meant "can't" :) I can't reproduce it. Aw well, I was surprised it was so easy :) yea, that would be too easy :) Can you try to upgrade to 1.8.12 ? A number of bugs have been fixed since I did try the upgrade to 1.8.12, got the same results (segfault) although I wasn't able to confirm it did segfault in the TCP splicing. What kind hove load do you have when it segfaults ? Far from enormous, maximum 10 requests per second, but as I said in my first post, the amount of TCP retransmissions and resets is very large due to the fact we're black-holing the traffic since we use haproxy for our captive portal I'd be happy to provide a pcap but for privacy reasons I can't extract it from a production environment and I can't see to replicate it in lab. I understand you don't want to send that kind of data. I definitively can't seem to reproduce it, with a configuration very similar to yours, including using netfilter to drop random packets to try to match your setup as best as possible. I'm afraid unless we're able to reproduce it, or you at least get a core, it'll be very difficult to debug. Regards, Olivier
Re: Issue with TCP splicing
Hi Julien, On Tue, Jul 24, 2018 at 01:29:49PM -0400, Julien Semaan wrote: > > Sorry, that was a "can" that really meant "can't" :) I can't reproduce it. > Aw well, I was surprised it was so easy :) > yea, that would be too easy :) > > Can you try to upgrade to 1.8.12 ? A number of bugs have been fixed since > I did try the upgrade to 1.8.12, got the same results (segfault) > although I wasn't able to confirm it did segfault in the TCP splicing. > > > What kind hove load do you have when it segfaults ? > Far from enormous, maximum 10 requests per second, but as I said in my > first post, the amount of TCP retransmissions and resets is very large due > to the fact we're black-holing the traffic since we use haproxy for our > captive portal > I'd be happy to provide a pcap but for privacy reasons I can't extract > it from a production environment and I can't see to replicate it in lab. > I understand you don't want to send that kind of data. I definitively can't seem to reproduce it, with a configuration very similar to yours, including using netfilter to drop random packets to try to match your setup as best as possible. I'm afraid unless we're able to reproduce it, or you at least get a core, it'll be very difficult to debug. Regards, Olivier
Re: Issue with TCP splicing
> Sorry, that was a "can" that really meant "can't" :) I can't reproduce it. Aw well, I was surprised it was so easy :) > Can you try to upgrade to 1.8.12 ? A number of bugs have been fixed since I did try the upgrade to 1.8.12, got the same results (segfault) although I wasn't able to confirm it did segfault in the TCP splicing. > What kind hove load do you have when it segfaults ? Far from enormous, maximum 10 requests per second, but as I said in my first post, the amount of TCP retransmissions and resets is very large due to the fact we're black-holing the traffic since we use haproxy for our captive portal I'd be happy to provide a pcap but for privacy reasons I can't extract it from a production environment and I can't see to replicate it in lab. Thanks! -- Julien Semaan jsem...@inverse.ca :: +1 (866) 353-6153 *155 ::www.inverse.ca Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org) On 2018-07-24 01:20 PM, Olivier Houchard wrote: Hi Julian, On Tue, Jul 24, 2018 at 12:58:27PM -0400, Julien Semaan wrote: Hi Olivier, Glad you're able to replicate it because I can't get it to happen consistently! I'd be happy if you could share the details of how it could be replicated if that's not too complex or hard to explain via email. Sorry, that was a "can" that really meant "can't" :) I can't reproduce it. Anyway, attached to this email, you'll find the haproxy configuration that has gotten the issue. Also, a lua script we do use and that is referenced in the configuration. This is running on CentOS 7.5.1804 with kernel 3.10.0-862.6.3.el7.x86_64 It is running through systemd, unit file attached to this email as well. Let me know if you need more infos, and I'll be glad to provide them or if you have a pcap I can replay to generate this issue in our lab. Best Regards, Thanks a lot for the informations ! I'm now pretty sure it's not related to directly splicing, more likely some memory corruption. Can you try to upgrade to 1.8.12 ? A number of bugs have been fixed since 1.8.9. What kind hove load do you have when it segfaults ? Thanks ! Olivier
Re: Issue with TCP splicing
Hi Julian, On Tue, Jul 24, 2018 at 12:58:27PM -0400, Julien Semaan wrote: > Hi Olivier, > > Glad you're able to replicate it because I can't get it to happen > consistently! > I'd be happy if you could share the details of how it could be replicated if > that's not too complex or hard to explain via email. Sorry, that was a "can" that really meant "can't" :) I can't reproduce it. > > Anyway, attached to this email, you'll find the haproxy configuration that > has gotten the issue. > Also, a lua script we do use and that is referenced in the configuration. > > This is running on CentOS 7.5.1804 with kernel 3.10.0-862.6.3.el7.x86_64 > > It is running through systemd, unit file attached to this email as well. > > Let me know if you need more infos, and I'll be glad to provide them or if > you have a pcap I can replay to generate this issue in our lab. > > Best Regards, > Thanks a lot for the informations ! I'm now pretty sure it's not related to directly splicing, more likely some memory corruption. Can you try to upgrade to 1.8.12 ? A number of bugs have been fixed since 1.8.9. What kind hove load do you have when it segfaults ? Thanks ! Olivier
Re: Issue with TCP splicing
Hi Olivier,
Glad you're able to replicate it because I can't get it to happen
consistently!
I'd be happy if you could share the details of how it could be
replicated if that's not too complex or hard to explain via email.
Anyway, attached to this email, you'll find the haproxy configuration
that has gotten the issue.
Also, a lua script we do use and that is referenced in the configuration.
This is running on CentOS 7.5.1804 with kernel 3.10.0-862.6.3.el7.x86_64
It is running through systemd, unit file attached to this email as well.
Let me know if you need more infos, and I'll be glad to provide them or
if you have a pcap I can replay to generate this issue in our lab.
Best Regards,
--
Julien Semaan
jsem...@inverse.ca :: +1 (866) 353-6153 *155 ::www.inverse.ca
Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence
(www.packetfence.org)
On 2018-07-24 12:28 PM, Olivier Houchard wrote:
Hi Julian,
On Mon, Jul 23, 2018 at 09:07:32AM -0400, Julien Semaan wrote:
Hi all,
We're currently using haproxy in our project PacketFence
(https://packetfence.org) and are currently experiencing an issue with
haproxy segfaulting when TCP splicing is enabled.
We're currently running version 1.8.9 and are occasionally getting segfaults
on this specific line in stream.c (line 2131):
(objt_cs(si_b->end) && __objt_cs(si_b->end)->conn->xprt &&
__objt_cs(si_b->end)->conn->xprt->snd_pipe) &&
I wasn't too bright when I found it through gdb and forgot to copy the
backtrace, so I'm hoping that the issue can be found with this limited
information.
After commenting out the code for TCP splicing with the patch attached to
the email, then the issue stopped happening.
Best Regards,
I can seem to reproduce this.
Care to share your configuration and your setup ?
Thanks !
Olivier
-- Update host on the fly
core.register_action("change_host", { 'http-req'}, function(txn)
if txn.sf:req_fhdr("Host") == nil then
txn.set_var(txn,"req.host","wireless.zammitcorp.ca") -- Update host on the fly
end
end)
-- Select backend based on Host header
local string = require("string");
core.register_action("select", { "http-req" }, function(txn)
if ( string.match(txn.sf:path(), '/profile.xml')) then
-- nothing, we let it go through for the XML profiles
elseif ( not ( string.match(txn.sf:req_fhdr("Host"):lower(), '^192%.168%.2%.10[0-9:]*$') or string.match(txn.sf:req_fhdr("Host"):lower(), '^192%.168%.4%.10[0-9:]*$') or string.match(txn.sf:req_fhdr("Host"):lower(), '^10%.61%.126%.10[0-9:]*$') or string.match(txn.sf:req_fhdr("Host"):lower(), '^wireless%.zammitcorp%.ca[0-9:]*$') or false ) ) then
txn:set_var("req.action","proxy")
else
if (txn.sf:path() == '/') then
txn:set_var("req.action","proxy")
elseif ( string.match(txn.sf:path(), '^/common') or string.match(txn.sf:path(), '^/content') or string.match(txn.sf:path(), '^/favicon.ico$') ) then
txn:set_var("req.action","static")
end
end
end)
global
external-check
user haproxy
group haproxy
daemon
pidfile /usr/local/pf/var/run/haproxy-portal.pid
log /dev/log local0
stats socket /tmp/proxystats level admin process 1
maxconn 4000
#Followup of https://github.com/inverse-inc/packetfence/pull/893
#haproxy 1.6.11 | intermediate profile | OpenSSL 1.0.1e | SRC:
https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy-1.6.11=1.0.1e=yes=intermediate
#Oldest compatible clients: Firefox 1, Chrome 1, IE 7, Opera 5, Safari
1, Windows XP IE8, Android 2.3, Java 7
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-bind-options no-sslv3 no-tls-tickets
ssl-default-server-ciphers
Re: Issue with TCP splicing
Hi Julian, On Mon, Jul 23, 2018 at 09:07:32AM -0400, Julien Semaan wrote: > Hi all, > > We're currently using haproxy in our project PacketFence > (https://packetfence.org) and are currently experiencing an issue with > haproxy segfaulting when TCP splicing is enabled. > > We're currently running version 1.8.9 and are occasionally getting segfaults > on this specific line in stream.c (line 2131): > (objt_cs(si_b->end) && __objt_cs(si_b->end)->conn->xprt && > __objt_cs(si_b->end)->conn->xprt->snd_pipe) && > > I wasn't too bright when I found it through gdb and forgot to copy the > backtrace, so I'm hoping that the issue can be found with this limited > information. > > After commenting out the code for TCP splicing with the patch attached to > the email, then the issue stopped happening. > > Best Regards, > I can seem to reproduce this. Care to share your configuration and your setup ? Thanks ! Olivier
Re: Issue with TCP splicing
Hi Julien. On 23/07/2018 13:59, Julien Semaan wrote: Doing it with the patch does the equivalent of disabling it with the option (realized there was an option afterwards). We're more looking to know if the haproxy team is interested in getting the issue addressed more than just getting the workaround From my experience with haproxy team I would say yes. Such cases are not really easy to find because the developer should be able to reproduce the behaviour, which can take some time and if in case it's special to your environment then there would be some amount of data and time from you and your team. As you can see on the mailing list archive there are a lot of long running and discussion threads to solve some specific and some common bugs ;-) https://www.mail-archive.com/haproxy@formilux.org/ As I'm just a member of the community and not a common developer I can only invite you to help us to solve this issue, I also want to tell you that I don't know how difficult or long running debug session this will be so let me please you to have some patience if it takes some time to debug and solve the issue. For starting it helps to know the full version, config and some " bt full " from core dumps from debug compiled version. One question from the enterprise vendor supported experience is. Can you produce the behaviour with the latest version ;-) Thanks! -- Julien Semaan Best regards Aleks jsem...@inverse.ca :: +1 (866) 353-6153 *155 ::www.inverse.ca Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org) On 2018-07-23 11:25 AM, Aleksandar Lazic wrote: Hi Julien. On 23/07/2018 09:07, Julien Semaan wrote: Hi all, We're currently using haproxy in our project PacketFence (https://packetfence.org) and are currently experiencing an issue with haproxy segfaulting when TCP splicing is enabled. We're currently running version 1.8.9 and are occasionally getting segfaults on this specific line in stream.c (line 2131): (objt_cs(si_b->end) && __objt_cs(si_b->end)->conn->xprt && __objt_cs(si_b->end)->conn->xprt->snd_pipe) && I wasn't too bright when I found it through gdb and forgot to copy the backtrace, so I'm hoping that the issue can be found with this limited information. After commenting out the code for TCP splicing with the patch attached to the email, then the issue stopped happening. Have you tried to disable splice via config? https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#nosplice Best Regards, -- Julien Semaan jsem...@inverse.ca :: +1 (866) 353-6153 *155 ::www.inverse.ca Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org) Best regards aleks
Re: Issue with TCP splicing
Doing it with the patch does the equivalent of disabling it with the option (realized there was an option afterwards). We're more looking to know if the haproxy team is interested in getting the issue addressed more than just getting the workaround Thanks! -- Julien Semaan jsem...@inverse.ca :: +1 (866) 353-6153 *155 ::www.inverse.ca Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org) On 2018-07-23 11:25 AM, Aleksandar Lazic wrote: Hi Julien. On 23/07/2018 09:07, Julien Semaan wrote: Hi all, We're currently using haproxy in our project PacketFence (https://packetfence.org) and are currently experiencing an issue with haproxy segfaulting when TCP splicing is enabled. We're currently running version 1.8.9 and are occasionally getting segfaults on this specific line in stream.c (line 2131): (objt_cs(si_b->end) && __objt_cs(si_b->end)->conn->xprt && __objt_cs(si_b->end)->conn->xprt->snd_pipe) && I wasn't too bright when I found it through gdb and forgot to copy the backtrace, so I'm hoping that the issue can be found with this limited information. After commenting out the code for TCP splicing with the patch attached to the email, then the issue stopped happening. Have you tried to disable splice via config? https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#nosplice Best Regards, -- Julien Semaan jsem...@inverse.ca :: +1 (866) 353-6153 *155 ::www.inverse.ca Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org) Best regards aleks
Re: Issue with TCP splicing
Hi Julien. On 23/07/2018 09:07, Julien Semaan wrote: Hi all, We're currently using haproxy in our project PacketFence (https://packetfence.org) and are currently experiencing an issue with haproxy segfaulting when TCP splicing is enabled. We're currently running version 1.8.9 and are occasionally getting segfaults on this specific line in stream.c (line 2131): (objt_cs(si_b->end) && __objt_cs(si_b->end)->conn->xprt && __objt_cs(si_b->end)->conn->xprt->snd_pipe) && I wasn't too bright when I found it through gdb and forgot to copy the backtrace, so I'm hoping that the issue can be found with this limited information. After commenting out the code for TCP splicing with the patch attached to the email, then the issue stopped happening. Have you tried to disable splice via config? https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#nosplice Best Regards, -- Julien Semaan jsem...@inverse.ca :: +1 (866) 353-6153 *155 ::www.inverse.ca Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org) Best regards aleks
Issue with TCP splicing
Hi all,
We're currently using haproxy in our project PacketFence
(https://packetfence.org) and are currently experiencing an issue with
haproxy segfaulting when TCP splicing is enabled.
We're currently running version 1.8.9 and are occasionally getting
segfaults on this specific line in stream.c (line 2131):
(objt_cs(si_b->end) && __objt_cs(si_b->end)->conn->xprt &&
__objt_cs(si_b->end)->conn->xprt->snd_pipe) &&
I wasn't too bright when I found it through gdb and forgot to copy the
backtrace, so I'm hoping that the issue can be found with this limited
information.
After commenting out the code for TCP splicing with the patch attached
to the email, then the issue stopped happening.
Best Regards,
--
Julien Semaan
jsem...@inverse.ca :: +1 (866) 353-6153 *155 ::www.inverse.ca
Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence
(www.packetfence.org)
diff -ruN haproxy-1.8.9.orig/src/stream.c haproxy-1.8.9/src/stream.c
--- haproxy-1.8.9.orig/src/stream.c 2018-05-18 09:10:29.0 -0400
+++ haproxy-1.8.9/src/stream.c 2018-07-20 13:06:41.861913134 -0400
@@ -2122,8 +2122,9 @@
if (s->txn)
s->txn->req.sov = s->txn->req.eoh + s->txn->req.eol - req->buf->o;
}
-
/* check if it is wise to enable kernel splicing to forward request data */
+ /* DON'T ENABLE TCP SPLICING AT ALL BECAUSE OF OCCASIONNAL SEGFAULTS WE'VE SEEN
+ * jsem...@inverse.ca
if (!(req->flags & (CF_KERN_SPLICING|CF_SHUTR)) &&
req->to_forward &&
(global.tune.options & GTUNE_USE_SPLICE) &&
@@ -2135,7 +2136,7 @@
(req->flags & CF_STREAMER_FAST {
req->flags |= CF_KERN_SPLICING;
}
-
+ */
/* reflect what the L7 analysers have seen last */
rqf_last = req->flags;
@@ -2306,6 +2307,8 @@
}
/* check if it is wise to enable kernel splicing to forward response data */
+ /* DON'T ENABLE TCP SPLICING AT ALL BECAUSE OF OCCASIONNAL SEGFAULTS WE'VE SEEN
+ * jsem...@inverse.ca
if (!(res->flags & (CF_KERN_SPLICING|CF_SHUTR)) &&
res->to_forward &&
(global.tune.options & GTUNE_USE_SPLICE) &&
@@ -2318,6 +2321,7 @@
res->flags |= CF_KERN_SPLICING;
}
+ */
/* reflect what the L7 analysers have seen last */
rpf_last = res->flags;
