Frietziek edited a comment on issue #13916:
URL: https://github.com/apache/airflow/issues/13916#issuecomment-830191383
Guys, I work with @alete89. Another solution for this, and specially if you
are in older Airflow versions that still don't have the
AIRFLOW__KUBERNETES__ENABLE_TCP_KEEPALIVE configuration key, is to execute at
some moment at the start of airflow this in a python script:
```Python
from urllib3.connection import HTTPConnection
import socket
HTTPConnection.default_socket_options =
HTTPConnection.default_socket_options + [
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 20),
(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 5),
(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 10)
]
```
This worked for us apparently, and basically set on urllib3 (which is the
library that airflow uses for connectivity under the hood) the same parametry
as was mentioned in this issue and in other places on the internet.
In our case, aparently, there were some tcp hangup that provoke the
consumption of all available executor capacity of parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
