voon created HUDI-7392:
--------------------------
Summary: Fix Connection leak causing CLOSE_WAIT TCP connections
Key: HUDI-7392
URL: https://issues.apache.org/jira/browse/HUDI-7392
Project: Apache Hudi
Issue Type: Bug
Reporter: voon
Assignee: voon
When consistent_hashing is enabled and a long running Spark job (Deltastreamer)
is created, we noticed that there is a gradual increase in CLOSE_WAIT
connections originating from the AM -> HDFS DN.
Command to check for close waits
{code:java}
netstat -anlpt | grep CLOSE_WAIT | grep 50010{code}
Result
{code:java}
tcp6 1 0 10.1.2.3:45994 10.5.4.3:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:48478 10.6.5.4:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:49542 10.7.6.5:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:47220 10.8.7.6:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:49786 10.9.8.7:50010 CLOSE_WAIT
2446/java {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
