Re: debugging ssl packet drop
Hi, I resent the mail since I didn't receive a copy of my first mail and I am subscribed to this list as well. In any case, I am not asking for help to debug any network issues here (if you read the content of my mail). There's an issue with a SSL packet being sent from IE10 browsers in the context of the websocket protocol (over ssl) and I have been working on the steps you mention further down in your email. My original mail was to see if I can get help regarding specific breakpoints in the code flow of the apache server/ssl module flow to check where the packet was being dropped. However, I'll take your suggestion that this probably needs to go to the openssl mailing list. Thank you. Regards, Umapathy On Fri, Oct 17, 2014 at 9:31 PM, Joe Lewis jle...@silverhawk.net wrote: Your first message was delivered less than 24 hours ago - most of us are not paid by the Apache modules developers list, meaning we are stricly volunteer, and 24 hours might not be enough time. I would suggest patience, especially while asking questions on the fringes of this lists expertise. Most people here are module developers, not SSL debuggers or TCP experts. I actually thought your original e-mail should have gone to an openssl mailing list instead of an Apache modules list. I won't support tracking down something like network errors without access and a consultation fee - it's an ugly rabbit hole that not many actually want to go down, especially me. I will simply suggest using Wireshark at multiple points (e.g. on the same LAN as the client, and on the same LAN as the server) just to ensure that a firewall, netscaler, or any other device between the client and the server isn't your problem. You claim an error code of 101 (network reset). Are you seeing TCP resets in your packet capture (remember, I'm not going to support or help beyond this - the question is to get you to think about what you are actually seeing in your packet capture). Did you decrypt your SSL-encrypted packet capture just to ensure you are seeing things properly? Are you sure you haven't custom-configured timeouts for Apache? ipfw/iptables/etc on the server (tcpdump of *nix will help)? Remember, it sounds like you are asking for help for things on the fringes of most people's expertise here. Be patient. Joe On Fri, Oct 17, 2014 at 9:22 AM, Pon Umapathy Kailash S pon.umapa...@gmail.com wrote: Resending since it doesn't seem to have been delivered. On Thu, Oct 16, 2014 at 11:26 AM, Pon Umapathy Kailash S pon.umapa...@gmail.com wrote: Hi, I am facing the an issue where a SSL packet from IE10 doesn't reach the client processing thread for a particular connection(more details below). Can you please provide me pointers on where to look/add more debug logs in the code to figure out what's happening? We use mpm worker threads. I have added support for websockets in a customised manner(as required for our application) inside apache. At a high level, it's done as follows: - the initial GET request with 101 code is handled by a handler hook function which computes the required security keys and sends back the response. Also, the socket on which the request came in is not closed(by maintaining a list and patching some parts of the apache code to not close if a socket is present in this list). - the child thread which processes this connection will relinquish the connection after the keep-alive timeout , which is ok since all we need is for the server to send messages to the client, with one exception. - At this point, the socket is recognised as a websocket client which is not yet authenticated(since from browsers we cannot set custom headers with the initial websocket connection request). - Authentication is done by the client sending the cookie as the 1st(and only) message on this connection to the server within the keep-alive timeout period(at which point the cookie is authenticated and the socket is marked as a valid, authenticated subscriber). (* there are other functions/timers to take care of stale, unauth connections etc) This works fine in all browsers with support for websockets with the following exceptions: IE10 over ssl(https/wss) and IE11 over ssl on 32-bit client machines. Doing a packet capture, we could figure out that the initial connection goes through fine and when the cookie is sent from the client, it reaches the server(and there's a tcp ack received at the client for this packet). However, the client processing this connection doesn't seem to receive this packet(this is well within the keep alive interval and the client thread is still actively processing that connection). Can you please let me know at which points in the code flow it might be useful to add debugging info to see where this is getting dropped? Regards, Umapathy
Re: debugging ssl packet drop
Another possible list thst would be good would be the HTTPD development list. You may need the author of mod_ssl. Thanks, Joe Lewis Original message From: Pon Umapathy Kailash S pon.umapa...@gmail.com Date: 10/18/2014 4:28 AM (GMT-07:00) To: modules-dev@httpd.apache.org Subject: Re: debugging ssl packet drop Hi, I resent the mail since I didn't receive a copy of my first mail and I am subscribed to this list as well. In any case, I am not asking for help to debug any network issues here (if you read the content of my mail). There's an issue with a SSL packet being sent from IE10 browsers in the context of the websocket protocol (over ssl) and I have been working on the steps you mention further down in your email. My original mail was to see if I can get help regarding specific breakpoints in the code flow of the apache server/ssl module flow to check where the packet was being dropped. However, I'll take your suggestion that this probably needs to go to the openssl mailing list. Thank you. Regards, Umapathy On Fri, Oct 17, 2014 at 9:31 PM, Joe Lewis jle...@silverhawk.net wrote: Your first message was delivered less than 24 hours ago - most of us are not paid by the Apache modules developers list, meaning we are stricly volunteer, and 24 hours might not be enough time. I would suggest patience, especially while asking questions on the fringes of this lists expertise. Most people here are module developers, not SSL debuggers or TCP experts. I actually thought your original e-mail should have gone to an openssl mailing list instead of an Apache modules list. I won't support tracking down something like network errors without access and a consultation fee - it's an ugly rabbit hole that not many actually want to go down, especially me. I will simply suggest using Wireshark at multiple points (e.g. on the same LAN as the client, and on the same LAN as the server) just to ensure that a firewall, netscaler, or any other device between the client and the server isn't your problem. You claim an error code of 101 (network reset). Are you seeing TCP resets in your packet capture (remember, I'm not going to support or help beyond this - the question is to get you to think about what you are actually seeing in your packet capture). Did you decrypt your SSL-encrypted packet capture just to ensure you are seeing things properly? Are you sure you haven't custom-configured timeouts for Apache? ipfw/iptables/etc on the server (tcpdump of *nix will help)? Remember, it sounds like you are asking for help for things on the fringes of most people's expertise here. Be patient. Joe On Fri, Oct 17, 2014 at 9:22 AM, Pon Umapathy Kailash S pon.umapa...@gmail.com wrote: Resending since it doesn't seem to have been delivered. On Thu, Oct 16, 2014 at 11:26 AM, Pon Umapathy Kailash S pon.umapa...@gmail.com wrote: Hi, I am facing the an issue where a SSL packet from IE10 doesn't reach the client processing thread for a particular connection(more details below). Can you please provide me pointers on where to look/add more debug logs in the code to figure out what's happening? We use mpm worker threads. I have added support for websockets in a customised manner(as required for our application) inside apache. At a high level, it's done as follows: - the initial GET request with 101 code is handled by a handler hook function which computes the required security keys and sends back the response. Also, the socket on which the request came in is not closed(by maintaining a list and patching some parts of the apache code to not close if a socket is present in this list). - the child thread which processes this connection will relinquish the connection after the keep-alive timeout , which is ok since all we need is for the server to send messages to the client, with one exception. - At this point, the socket is recognised as a websocket client which is not yet authenticated(since from browsers we cannot set custom headers with the initial websocket connection request). - Authentication is done by the client sending the cookie as the 1st(and only) message on this connection to the server within the keep-alive timeout period(at which point the cookie is authenticated and the socket is marked as a valid, authenticated subscriber). (* there are other functions/timers to take care of stale, unauth connections etc) This works fine in all browsers with support for websockets with the following exceptions: IE10 over ssl(https/wss) and IE11 over ssl on 32-bit client machines. Doing a packet capture, we could figure out that the initial connection goes through fine and when the cookie is sent from the client, it reaches the server(and there's a tcp ack received at the client for this packet). However, the client processing this connection doesn't seem to receive this packet(this is well
Re: svn commit: r1632736 - in /httpd/httpd/branches/2.4.x: CHANGES STATUS modules/proxy/mod_proxy_http.c
Hi, Isn't there the same kind of potential issue in: mod_buffer, line 268 mod_cahce, line 687 Best regards, CJ Le 18/10/2014 08:57, jaillet...@apache.org a écrit : Author: jailletc36 Date: Sat Oct 18 06:57:40 2014 New Revision: 1632736 URL: http://svn.apache.org/r1632736 Log: Merge r1599486 from trunk mod_proxy_http: Avoid (unlikely) access to freed memory. Submitted by: ylavic Reviewed by: ylavic, jorton, rjung Backported by: jailletc36 Modified: httpd/httpd/branches/2.4.x/CHANGES httpd/httpd/branches/2.4.x/STATUS httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_http.c Modified: httpd/httpd/branches/2.4.x/CHANGES URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/CHANGES?rev=1632736r1=1632735r2=1632736view=diff == --- httpd/httpd/branches/2.4.x/CHANGES [utf-8] (original) +++ httpd/httpd/branches/2.4.x/CHANGES [utf-8] Sat Oct 18 06:57:40 2014 @@ -13,6 +13,8 @@ Changes with Apache 2.4.11 request headers earlier. Adds MergeTrailers directive to restore legacy behavior. [Edward Lu, Yann Ylavic, Joe Orton, Eric Covener] + *) mod_proxy_http: Avoid (unlikely) access to freed memory. [Yann Ylavic] + *) http_protocol: fix logic in ap_method_list_(add|remove) in order: - to correctly reset bits - not to modify the 'method_mask' bitfield unnecessarily Modified: httpd/httpd/branches/2.4.x/STATUS URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/STATUS?rev=1632736r1=1632735r2=1632736view=diff == --- httpd/httpd/branches/2.4.x/STATUS (original) +++ httpd/httpd/branches/2.4.x/STATUS Sat Oct 18 06:57:40 2014 @@ -108,19 +108,6 @@ PATCHES ACCEPTED TO BACKPORT FROM TRUNK: 2.4.x patch: trunk works +1: jkaluza, ylavic, rjung - * mod_proxy_http: Avoid (unlikely) access to freed memory. - trunk patch: http://svn.apache.org/r1599486 - 2.4.x patch: trunk works - +1: ylavic, jorton, rjung (as is) - covener: I did not look in depth, but is the preceding log message also bad? - ylavic: No, this concerns the next for (;; e = APR_BUCKET_NEXT(e)) iteration. - We could also s/apr_bucket_delete/APR_BUCKET_REMOVE/ instead, but - stripping some (unhandled) buckets from the source brigade does - not look correct to me either (brigade *to is to be consumed, but - *from is still living, the caller may want to reuse it, eg: - https://issues.apache.org/bugzilla/attachment.cgi?id=31686). - Should we? - * mod_proxy: Make worker name truncation a non-fatal error. trunk patch: http://svn.apache.org/r1621367 http://svn.apache.org/r1621372 Modified: httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_http.c URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_http.c?rev=1632736r1=1632735r2=1632736view=diff == --- httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_http.c (original) +++ httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_http.c Sat Oct 18 06:57:40 2014 @@ -687,7 +687,6 @@ static apr_status_t proxy_buckets_lifeti ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, APLOGNO(00964) Unhandled bucket type of type %s in proxy_buckets_lifetime_transform, e-type-name); -apr_bucket_delete(e); rv = APR_EGENERAL; } }
Re: svn commit: r1632736 - in /httpd/httpd/branches/2.4.x: CHANGES STATUS modules/proxy/mod_proxy_http.c
Hi Christophe, On Sat, Oct 18, 2014 at 9:20 AM, Marion Christophe JAILLET christophe.jail...@wanadoo.fr wrote: Hi, Isn't there the same kind of potential issue in: mod_buffer, line 268 mod_cahce, line 687 Yes both, good catches. Do you want/plan to fix them? Regards, Yann.
Re: svn commit: r1632736 - in /httpd/httpd/branches/2.4.x: CHANGES STATUS modules/proxy/mod_proxy_http.c
Done in r1632740 and r1632742. Backport proposed in r1632744. Thanks, Yann. On Sat, Oct 18, 2014 at 9:28 AM, Yann Ylavic ylavic@gmail.com wrote: Hi Christophe, On Sat, Oct 18, 2014 at 9:20 AM, Marion Christophe JAILLET christophe.jail...@wanadoo.fr wrote: Hi, Isn't there the same kind of potential issue in: mod_buffer, line 268 mod_cahce, line 687 Yes both, good catches. Do you want/plan to fix them? Regards, Yann.
RE: Listeners buckets and duplication w/ and w/o SO_REUSEPORT on trunk
Hi All, I just want to check if there is any feedback on this? Generated based on trunk version, this is to remove some code duplications/global variables. This also removes listener duplication when SO_REUSEPORT is not being used. For details, please refer to Yann Ylavic's notes and my responses below. I also attached the code changes here again in case you missed it in the original email I sent last Friday. Thanks very much, Yingqi -Original Message- From: Lu, Yingqi [mailto:yingqi...@intel.com] Sent: Friday, October 10, 2014 4:56 PM To: dev@httpd.apache.org Subject: RE: Listeners buckets and duplication w/ and w/o SO_REUSEPORT on trunk Dear All, Attached patch is generated based on current trunk. It covers for prefork/worker/event/eventopt MPM. It supposes to address following issues regarding to SO_RESUEPORT support vs. current trunk version: 1. Same as current trunk version implementation, when active_CPU_num = 8 or when so_reuseport is not supported by the kernel, ap_num_buckets is set to 1. In any case, there is 1 dedicated listener per bucket. 2. Remove global variables (mpm_listen, enable_default_listeners and num_buckets). mpm_listen is changed to MPM local. enabled_default_listener is completely removed. num_buckets is changed to MPM local (ap_num_buckets). I rename have_so_reuseport to ap_have_so_reuseport. The reason for keeping that global is because this variable is being used by ap_log_common(). Based on the feedback here, I think it may not be a good idea to change the function interface. 3. Change ap_duplicated_listener to have more parameters. This function is being called from MPM local (prefork.c/worker.c/event.c/eventopt.c). In this function, prefork_listener (or worker_listener/even_listener/etc) array will be initialized and be set value. ap_num_buckets is also calculated inside this function. In addition, this version solves the issue with one_process case (current trunk version has issue with one_process enabled). 4. Change ap_close_listener() back to previous (2.4.X version). 5. Change dummy_connection back to previous (2.4.X version). 6. Add ap_close_duplicated_listeners(). This is called from mpms when stopping httpd. 7. Add ap_close_child_listener(). When listener_thread (child process in prefork) exit, only the dedicated listener needs to be closed (the rest are already being closed in child_main when the child process starts). 8. Remove duplication of listener when ap_num_buckets = 1 or without SO_REUSEPORT support (ap_num_buckets = 1). With so_reuseport, only duplicated (ap_num_buckets - 1) listeners (1 less duplication less current trunk implementation). 9. Inside each mpm, move child_bucket, child_pod and child_mutex (worker/prefork only) to a struct. Also, add member bucket to the same struct. Please review and let me know your feedback. Thanks, Yingqi -Original Message- From: Yann Ylavic [mailto:ylavic@gmail.com] Sent: Tuesday, October 07, 2014 5:26 PM To: httpd Subject: Re: Listeners buckets and duplication w/ and w/o SO_REUSEPORT on trunk On Wed, Oct 8, 2014 at 2:03 AM, Yann Ylavic ylavic@gmail.com wrote: On Wed, Oct 8, 2014 at 1:50 AM, Lu, Yingqi yingqi...@intel.com wrote: 3. Yes, I did use some extern variables. I can change the name of them to better coordinate with the variable naming conversion. We should do something with ap_prefixed? Is there anything else I should consider when I rename the variable? Maybe defining new functions with more arguments (to be used by the existing ones with NULL or default values) is a better alternative. For example, ap_duplicate_listeners could be modified to provide mpm_listen and even do the computation of num_buckets and provide it (this is not an API change since it is trunk only for now). ap_close_listeners() could be then restored as before (work on ap_listeners only) and ap_close_duplicated_listeners(mpm_listen) be introduced and used in the MPMs instead. Hence ap_listen_rec *mpm_listeners could be MPM local, which would then call ap_duplicate_listeners(..., mpm_listeners, num_buckets) and ap_close_duplicated_listeners(mpm_listeners) That's just a quick thought... Please be aware that existing AP_DECLAREd functions API must not change though. Regards, Yann. Thanks, Yingqi -Original Message- From: Yann Ylavic [mailto:ylavic@gmail.com] Sent: Tuesday, October 07, 2014 4:19 PM To: httpd Subject: Listeners buckets and duplication w/ and w/o SO_REUSEPORT on trunk Hi, some notes about the current implementation of this (trunk only). First, whether or not SO_REUSEPORT is available, we do duplicate the listeners. This, I think, is not the intention of Yingqi Lu's original proposal, and probably my fault since I asked for the patch to be splitted in two for a better understanding (finally the SO_REUSEPORT patch only has been commited). The fact is that without SO_REUSEPORT, this serves
