[
https://issues.apache.org/jira/browse/YARN-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tarun Parimi updated YARN-9336:
-------------------------------
Description:
The JobHistoryServer is leaking CLOSE_WAIT connections to DataNodes whenever
viewing a huge log file in JobhistoryServer. This happens only when the below
is configured.
{code:java}
yarn.log-aggregation.file-formats=IndexedFormat
yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs
{code}
On investigation, I found that the FSDataInputStream is not closed in
org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock
. Since this block is called every time the Jobhistory page displays the logs,
CLOSE_WAIT connections to DataNodes keep on increasing in JobHistoryServer.
was:
The JobHistoryServer is leaking CLOSE_WAIT connections whenever viewing a huge
log file in JobhistoryServer. This happens only when the below is configured.
{code:java}
yarn.log-aggregation.file-formats=IndexedFormat
yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs
{code}
On investigation, I found that the FSDataInputStream is not closed in
org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock
. Since this block is called every time the Josbhistory page displays the logs
CLOSE_WAIT connections to datanode keep on increasing in JobHistoryServer.
> JobHistoryServer leaks CLOSE_WAIT tcp connections when using
> LogAggregationIndexedFileController
> ------------------------------------------------------------------------------------------------
>
> Key: YARN-9336
> URL: https://issues.apache.org/jira/browse/YARN-9336
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 2.9.0
> Reporter: Tarun Parimi
> Assignee: Tarun Parimi
> Priority: Major
> Attachments: YARN-9336.001.patch
>
>
> The JobHistoryServer is leaking CLOSE_WAIT connections to DataNodes whenever
> viewing a huge log file in JobhistoryServer. This happens only when the below
> is configured.
> {code:java}
> yarn.log-aggregation.file-formats=IndexedFormat
> yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
> yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs
> {code}
> On investigation, I found that the FSDataInputStream is not closed in
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock
> . Since this block is called every time the Jobhistory page displays the
> logs, CLOSE_WAIT connections to DataNodes keep on increasing in
> JobHistoryServer.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
