Re: (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted
On 2/12/25 2:21 PM, Yann Ylavic wrote: On Wed, Feb 12, 2025 at 12:38 PM Yann Ylavic wrote: However it seems that [4] will cause worker->s->is_address_reusable = 0 in [3] (i.e. disable DNS reuse altogether) which is not really expected, and would explain why the pool is cleared in connection_cleanup(). [3] https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L1989-L2034 [4] https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L2140-L2159 I think we need something like the attached patch to not let the ->disablereuse=1 in [4] (because of a $-substitution including outside hostname[:port]) force ->is_address_reusable=0 in [3] implicitly. The rationale is that before 2.4.59 ->is_address_reusable and ->disablereuse were kind of the same thing (the former, not configurable, was set from the latter), but since 2.4.59 ->is_address_reusable is about the reusability of backend's DNS address (and lifetime, both deduced from the address_ttl= parameter) while ->disablereuse is about the reusability of the connections/sockets only. the mod_cluster code does: worker->s->is_address_reusable = 1; and doesn't care about worker->s->disablereuse... Except we probably still want to disable address reuse when disablereuse=on explicitly (for compatibility), but not for the implicit case [4], which is what this patch does. Thoughts? Jean-Frederic, I don't think it addresses your issue because you probably have enablereuse=on already if your ProxyPassMatch used to reuse connections before 2.4.59? Connection reuse for ProxyPassMatch is possible since 2.4.47, but has always depended on enablereuse=on. If you don't use ProxyPassMatch with a $-substitution I don't see why something changed in 2.4.59 though.. I guess I have to fix mod_cluster, thanks for help. -- Cheers Jean-Frederic
Re: (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted
On Wed, Feb 12, 2025 at 12:38 PM Yann Ylavic wrote:
>
> However it seems that [4] will cause worker->s->is_address_reusable =
> 0 in [3] (i.e. disable DNS reuse altogether) which is not really
> expected, and would explain why the pool is cleared in
> connection_cleanup().
>
> [3]
> https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L1989-L2034
> [4]
> https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L2140-L2159
I think we need something like the attached patch to not let the
->disablereuse=1 in [4] (because of a $-substitution including outside
hostname[:port]) force ->is_address_reusable=0 in [3] implicitly.
The rationale is that before 2.4.59 ->is_address_reusable and
->disablereuse were kind of the same thing (the former, not
configurable, was set from the latter), but since 2.4.59
->is_address_reusable is about the reusability of backend's DNS
address (and lifetime, both deduced from the address_ttl= parameter)
while ->disablereuse is about the reusability of the
connections/sockets only.
Except we probably still want to disable address reuse when
disablereuse=on explicitly (for compatibility), but not for the
implicit case [4], which is what this patch does.
Thoughts?
Jean-Frederic, I don't think it addresses your issue because you
probably have enablereuse=on already if your ProxyPassMatch used to
reuse connections before 2.4.59? Connection reuse for ProxyPassMatch
is possible since 2.4.47, but has always depended on enablereuse=on.
If you don't use ProxyPassMatch with a $-substitution I don't see why
something changed in 2.4.59 though..
Index: modules/proxy/proxy_util.c
===
--- modules/proxy/proxy_util.c (revision 1923126)
+++ modules/proxy/proxy_util.c (working copy)
@@ -2262,12 +2262,18 @@ PROXY_DECLARE(apr_status_t) ap_proxy_initialize_wo
if (!worker->s->retry_set) {
worker->s->retry = apr_time_from_sec(PROXY_WORKER_DEFAULT_RETRY);
}
-/* Consistently set address and connection reusabilty: when reuse
- * is disabled by configuration, or when the address is known already
- * to not be reusable for this worker (in any case, thus ignore/force
- * DisableReuse).
+/* worker->s->disablereuse is about disabling backend connection/socket
+ * reuse for successive requests, while worker->s->is_address_reusable
+ * is about DNS address reuse (an worker->s->address_ttl of zero also
+ * means not reusable). When the DNS address is not reusable neither
+ * are the connections, so make this consistent here.
+ * However setting disablereuse=on (or enablereuse=off) has always
+ * disabled DNS address reuse too, both were kind of the same thing
+ * before 2.4.59, so we keep this behaviour unless an address_ttl= is
+ * configured explicitely.
*/
if (!worker->s->address_ttl || (!worker->s->address_ttl_set
+&& worker->s->disablereuse_set
&& worker->s->disablereuse)) {
worker->s->is_address_reusable = 0;
}
Re: (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted
On Wed, Feb 12, 2025 at 12:07 PM Yann Ylavic wrote: > > On Wed, Feb 12, 2025 at 11:16 AM jean-frederic clere > wrote: > > > > According to my tests for some reasons ap_proxy_release_connection() now > > does an apr_pool_clear() and was doing a connection_cleanup() before 2.4.58. > > What does your ProxyPass/ProxyPassMatch look like? > AFAICT connection_cleanup() closes the connection by clearing its pool > only if !worker->s->is_address_reusable ([1]), meaning disablereuse=on > or enablereuse=off was set ([2]), or the ProxyPassMatch contains a > $-substitution in the hostname[:port] part of the URL ([3]). > > [1] > https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L1657-L1661 > [2] > https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L2270-L2273 > [3] > https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L1989-L2034 Oh, and there is also this ProxyPassMatch case ([4]) where we force disablereuse for compatibility with < 2.4.47 which never reused connections. So if there is a $-substitution *anywhere* in the worker URL then connection reuse is disabled by default, but it can still be enabled explicitly with enablereuse=on. However it seems that [4] will cause worker->s->is_address_reusable = 0 in [3] (i.e. disable DNS reuse altogether) which is not really expected, and would explain why the pool is cleared in connection_cleanup(). Though whether the pool is cleared (!is_address_reusable) or the connection only is closed (disablereuse) shouldn't make a difference for your issue, in both cases the socket gets closed for each request.. [4] https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L2140-L2159 > > But it seems that mod_proxy_cluster is creating and initializing its > own proxy workers, and I don't know which games it plays on its own > with worker->s->disablereuse or worker->s->is_address_reusable or > worker->s->is_address_reusable. > > > Regards; > Yann.
Re: (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted
On Wed, Feb 12, 2025 at 11:16 AM jean-frederic clere wrote: > > According to my tests for some reasons ap_proxy_release_connection() now > does an apr_pool_clear() and was doing a connection_cleanup() before 2.4.58. What does your ProxyPass/ProxyPassMatch look like? AFAICT connection_cleanup() closes the connection by clearing its pool only if !worker->s->is_address_reusable ([1]), meaning disablereuse=on or enablereuse=off was set ([2]), or the ProxyPassMatch contains a $-substitution in the hostname[:port] part of the URL ([3]). [1] https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L1657-L1661 [2] https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L2270-L2273 [3] https://github.com/apache/httpd/blob/2.4.63/modules/proxy/proxy_util.c#L1989-L2034 But it seems that mod_proxy_cluster is creating and initializing its own proxy workers, and I don't know which games it plays on its own with worker->s->disablereuse or worker->s->is_address_reusable or worker->s->is_address_reusable. Regards; Yann.
Re: (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted
On 2/11/25 2:24 PM, Yann Ylavic wrote: On Fri, Feb 7, 2025 at 5:56 PM jean-frederic clere wrote: I have the above error on windows: +++ [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] mod_proxy.c(1465): [client ::1:56280] AH01143: Running scheme balancer handler (attempt 0) [Fri Feb 07 08:46:33.110167 2025] [proxy_ajp:debug] [pid 7440:tid 1168] mod_proxy_ajp.c(790): [client ::1:56280] AH00895: serving URL ajp://[0:0:0:0:0:0:0:1]/ [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] proxy_util.c(2797): AH00942: AJP: has acquired connection for (0:0:0:0:0:0:0:1:8009) [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] proxy_util.c(3242): [client ::1:56280] AH00944: connecting ajp://[0:0:0:0:0:0:0:1]/ to 0:0:0:0:0:0:0:1:8009 [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] proxy_util.c(3450): [client ::1:56280] AH00947: connecting / to [::1]:8009 (0:0:0:0:0:0:0:1:8009) [Fri Feb 07 08:46:33.110167 2025] [proxy:error] [pid 7440:tid 2552] (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : AH00957: AJP: attempt to connect to [::1]:8009 (0:0:0:0:0:0:0:1:8009) failed [Fri Feb 07 08:46:33.110167 2025] [proxy:error] [pid 7440:tid 2552] AH00959: ap_proxy_connect_backend disabling worker for (0:0:0:0:0:0:0:1:8009) for 60s [Fri Feb 07 08:46:33.110167 2025] [proxy_ajp:error] [pid 7440:tid 2552] [client ::1:49653] AH00896: failed to make connection to backend: 0:0:0:0:0:0:0:1 [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 2552] proxy_util.c(2813): AH00943: AJP: has released connection for (0:0:0:0:0:0:0:1:8009) +++ Does that ring bell to someone? I am using 2.4.x and my own build on windows (windows server 2019) "(OS 10048)" seems to be the "ephemeral ports exhausted" error on Windows, so maybe [0] can help? According to my tests for some reasons ap_proxy_release_connection() now does an apr_pool_clear() and was doing a connection_cleanup() before 2.4.58. [0] https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/default-dynamic-port-range-tcpip-chang helps Regards; Yann. -- Cheers Jean-Frederic
Re: (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted
On 2/11/25 2:24 PM, Yann Ylavic wrote: On Fri, Feb 7, 2025 at 5:56 PM jean-frederic clere wrote: I have the above error on windows: +++ [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] mod_proxy.c(1465): [client ::1:56280] AH01143: Running scheme balancer handler (attempt 0) [Fri Feb 07 08:46:33.110167 2025] [proxy_ajp:debug] [pid 7440:tid 1168] mod_proxy_ajp.c(790): [client ::1:56280] AH00895: serving URL ajp://[0:0:0:0:0:0:0:1]/ [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] proxy_util.c(2797): AH00942: AJP: has acquired connection for (0:0:0:0:0:0:0:1:8009) [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] proxy_util.c(3242): [client ::1:56280] AH00944: connecting ajp://[0:0:0:0:0:0:0:1]/ to 0:0:0:0:0:0:0:1:8009 [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] proxy_util.c(3450): [client ::1:56280] AH00947: connecting / to [::1]:8009 (0:0:0:0:0:0:0:1:8009) [Fri Feb 07 08:46:33.110167 2025] [proxy:error] [pid 7440:tid 2552] (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : AH00957: AJP: attempt to connect to [::1]:8009 (0:0:0:0:0:0:0:1:8009) failed [Fri Feb 07 08:46:33.110167 2025] [proxy:error] [pid 7440:tid 2552] AH00959: ap_proxy_connect_backend disabling worker for (0:0:0:0:0:0:0:1:8009) for 60s [Fri Feb 07 08:46:33.110167 2025] [proxy_ajp:error] [pid 7440:tid 2552] [client ::1:49653] AH00896: failed to make connection to backend: 0:0:0:0:0:0:0:1 [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 2552] proxy_util.c(2813): AH00943: AJP: has released connection for (0:0:0:0:0:0:0:1:8009) +++ Does that ring bell to someone? I am using 2.4.x and my own build on windows (windows server 2019) "(OS 10048)" seems to be the "ephemeral ports exhausted" error on Windows, so maybe [0] can help? Yes that is the problem, but the cause is somewhere in httpd proxy. When running a test using ab with my module (mod_cluster that uses mod_proxy API), it passes with 2.4.58 and fails with 2.4.59. There many more sockets in TIME_WAIT (2 vs 15000) in 2.4.59 +++ $ netstat -n | grep TIME_WAIT | wc -l 21620 +++ Something in mod_proxy API changed and causes the problem, I need to figure out what and how to call it the right way. [0] https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/default-dynamic-port-range-tcpip-chang helps Regards; Yann. -- Cheers Jean-Frederic
Re: (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted
On Fri, Feb 7, 2025 at 5:56 PM jean-frederic clere wrote: > > I have the above error on windows: > +++ > [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] > mod_proxy.c(1465): [client ::1:56280] AH01143: Running scheme balancer > handler (attempt 0) > [Fri Feb 07 08:46:33.110167 2025] [proxy_ajp:debug] [pid 7440:tid 1168] > mod_proxy_ajp.c(790): [client ::1:56280] AH00895: serving URL > ajp://[0:0:0:0:0:0:0:1]/ > [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] > proxy_util.c(2797): AH00942: AJP: has acquired connection for > (0:0:0:0:0:0:0:1:8009) > [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] > proxy_util.c(3242): [client ::1:56280] AH00944: connecting > ajp://[0:0:0:0:0:0:0:1]/ to 0:0:0:0:0:0:0:1:8009 > [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] > proxy_util.c(3450): [client ::1:56280] AH00947: connecting / to > [::1]:8009 (0:0:0:0:0:0:0:1:8009) > [Fri Feb 07 08:46:33.110167 2025] [proxy:error] [pid 7440:tid 2552] (OS > 10048)Only one usage of each socket address (protocol/network > address/port) is normally permitted. : AH00957: AJP: attempt to connect > to [::1]:8009 (0:0:0:0:0:0:0:1:8009) failed > [Fri Feb 07 08:46:33.110167 2025] [proxy:error] [pid 7440:tid 2552] > AH00959: ap_proxy_connect_backend disabling worker for > (0:0:0:0:0:0:0:1:8009) for 60s > [Fri Feb 07 08:46:33.110167 2025] [proxy_ajp:error] [pid 7440:tid 2552] > [client ::1:49653] AH00896: failed to make connection to backend: > 0:0:0:0:0:0:0:1 > [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 2552] > proxy_util.c(2813): AH00943: AJP: has released connection for > (0:0:0:0:0:0:0:1:8009) > +++ > > Does that ring bell to someone? I am using 2.4.x and my own build on > windows (windows server 2019) "(OS 10048)" seems to be the "ephemeral ports exhausted" error on Windows, so maybe [0] can help? [0] https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/default-dynamic-port-range-tcpip-chang helps Regards; Yann.
(OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted
Hi, I have the above error on windows: +++ [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] mod_proxy.c(1465): [client ::1:56280] AH01143: Running scheme balancer handler (attempt 0) [Fri Feb 07 08:46:33.110167 2025] [proxy_ajp:debug] [pid 7440:tid 1168] mod_proxy_ajp.c(790): [client ::1:56280] AH00895: serving URL ajp://[0:0:0:0:0:0:0:1]/ [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] proxy_util.c(2797): AH00942: AJP: has acquired connection for (0:0:0:0:0:0:0:1:8009) [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] proxy_util.c(3242): [client ::1:56280] AH00944: connecting ajp://[0:0:0:0:0:0:0:1]/ to 0:0:0:0:0:0:0:1:8009 [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 1168] proxy_util.c(3450): [client ::1:56280] AH00947: connecting / to [::1]:8009 (0:0:0:0:0:0:0:1:8009) [Fri Feb 07 08:46:33.110167 2025] [proxy:error] [pid 7440:tid 2552] (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : AH00957: AJP: attempt to connect to [::1]:8009 (0:0:0:0:0:0:0:1:8009) failed [Fri Feb 07 08:46:33.110167 2025] [proxy:error] [pid 7440:tid 2552] AH00959: ap_proxy_connect_backend disabling worker for (0:0:0:0:0:0:0:1:8009) for 60s [Fri Feb 07 08:46:33.110167 2025] [proxy_ajp:error] [pid 7440:tid 2552] [client ::1:49653] AH00896: failed to make connection to backend: 0:0:0:0:0:0:0:1 [Fri Feb 07 08:46:33.110167 2025] [proxy:debug] [pid 7440:tid 2552] proxy_util.c(2813): AH00943: AJP: has released connection for (0:0:0:0:0:0:0:1:8009) +++ Does that ring bell to someone? I am using 2.4.x and my own build on windows (windows server 2019) -- Cheers Jean-Frederic
