[
https://issues.apache.org/jira/browse/HUDI-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
voon updated HUDI-7392:
-----------------------
Description:
When consistent_hashing is enabled and a long running Spark job (Deltastreamer)
is created, we noticed that there is a gradual increase in CLOSE_WAIT
connections originating from the AM -> HDFS DN.
Command to check for close waits
{code:java}
netstat -anlpt | grep CLOSE_WAIT | grep 50010{code}
Result
{code:java}
tcp6 1 0 10.1.2.3:45994 10.5.4.3:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:48478 10.6.5.4:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:49542 10.7.6.5:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:47220 10.8.7.6:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:49786 10.9.8.7:50010 CLOSE_WAIT
2446/java {code}
To reproduce this:
{code:java}
CREATE TABLE dev_hudi.close_wait_issue_investigation (
id INT,
name STRING,
date_col STRING,
grass_region STRING
) USING hudi
PARTITIONED BY (grass_region)
tblproperties (
primaryKey = 'id',
type = 'mor',
precombineField = 'id',
hoodie.index.type = 'BUCKET',
hoodie.index.bucket.engine = 'CONSISTENT_HASHING',
hoodie.compact.inline = 'true'
)
LOCATION 'hdfs://DEV/close_wait_issue_investigation';
INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (1, 'alex1',
'2023-12-22', 'SG');
INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (2, 'alex2',
'2023-12-22', 'SG');
INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (3, 'alex3',
'2023-12-22', 'SG');
INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (4, 'alex4',
'2023-12-22', 'SG');
INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (5, 'alex5',
'2023-12-22', 'SG');{code}
was:
When consistent_hashing is enabled and a long running Spark job (Deltastreamer)
is created, we noticed that there is a gradual increase in CLOSE_WAIT
connections originating from the AM -> HDFS DN.
Command to check for close waits
{code:java}
netstat -anlpt | grep CLOSE_WAIT | grep 50010{code}
Result
{code:java}
tcp6 1 0 10.1.2.3:45994 10.5.4.3:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:48478 10.6.5.4:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:49542 10.7.6.5:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:47220 10.8.7.6:50010 CLOSE_WAIT
2446/java
tcp6 1 0 10.1.2.3:49786 10.9.8.7:50010 CLOSE_WAIT
2446/java {code}
> Fix Connection leak causing CLOSE_WAIT TCP connections
> ------------------------------------------------------
>
> Key: HUDI-7392
> URL: https://issues.apache.org/jira/browse/HUDI-7392
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: voon
> Assignee: voon
> Priority: Major
>
> When consistent_hashing is enabled and a long running Spark job
> (Deltastreamer) is created, we noticed that there is a gradual increase in
> CLOSE_WAIT connections originating from the AM -> HDFS DN.
>
> Command to check for close waits
> {code:java}
> netstat -anlpt | grep CLOSE_WAIT | grep 50010{code}
> Result
> {code:java}
> tcp6 1 0 10.1.2.3:45994 10.5.4.3:50010 CLOSE_WAIT
> 2446/java
> tcp6 1 0 10.1.2.3:48478 10.6.5.4:50010 CLOSE_WAIT
> 2446/java
> tcp6 1 0 10.1.2.3:49542 10.7.6.5:50010 CLOSE_WAIT
> 2446/java
> tcp6 1 0 10.1.2.3:47220 10.8.7.6:50010 CLOSE_WAIT
> 2446/java
> tcp6 1 0 10.1.2.3:49786 10.9.8.7:50010 CLOSE_WAIT
> 2446/java {code}
>
> To reproduce this:
>
> {code:java}
> CREATE TABLE dev_hudi.close_wait_issue_investigation (
> id INT,
> name STRING,
> date_col STRING,
> grass_region STRING
> ) USING hudi
> PARTITIONED BY (grass_region)
> tblproperties (
> primaryKey = 'id',
> type = 'mor',
> precombineField = 'id',
> hoodie.index.type = 'BUCKET',
> hoodie.index.bucket.engine = 'CONSISTENT_HASHING',
> hoodie.compact.inline = 'true'
> )
> LOCATION 'hdfs://DEV/close_wait_issue_investigation';
> INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (1, 'alex1',
> '2023-12-22', 'SG');
> INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (2, 'alex2',
> '2023-12-22', 'SG');
> INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (3, 'alex3',
> '2023-12-22', 'SG');
> INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (4, 'alex4',
> '2023-12-22', 'SG');
> INSERT INTO dev_hudi.close_wait_issue_investigation VALUES (5, 'alex5',
> '2023-12-22', 'SG');{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
