[
https://issues.apache.org/jira/browse/AMBARI-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-17834:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
committed to trunk:
{code:java}
commit 842d382efc98bd1264b618cd42cad3c5f5b424c0
Author: oleewere <[email protected]>
Date: Sun Jul 24 04:07:14 2016 +0200
AMBARI-17834. Logfeeder: Removed file that is processed by hdfsCopyThread
from readylist (Hayat Behlim via oleewere)
{code}
comitted to branch-2.4:
{code:java}
commit d65072df1084a19c89d7d929399c227929c09585
Author: oleewere <[email protected]>
Date: Sun Jul 24 04:07:14 2016 +0200
AMBARI-17834. Logfeeder: Removed file that is processed by hdfsCopyThread
from readylist (Hayat Behlim via oleewere)
{code}
> HDFS copy thread in OutputHDFSFile is not removing files that are processed
> ----------------------------------------------------------------------------
>
> Key: AMBARI-17834
> URL: https://issues.apache.org/jira/browse/AMBARI-17834
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
> Affects Versions: 2.4.0
> Reporter: Hemanth Yamijala
> Assignee: Hayat Behlim
> Fix For: 2.4.0
>
> Attachments: AMBARI-17834.patch
>
>
> Consider the HDFS copy thread here:
> https://github.com/apache/ambari/blob/trunk/ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputHDFSFile.java#L177
> It uses a data structure localReadyFiles that has the list of files to copy
> to HDFS. But AFAICS, nowhere are processed files removed from this data
> structure. This could result in multiple incorrect behaviours:
> * Over a period of time, this structure could keep growing indefinitely.
> While the check for existence of files will allow progress, it will take
> longer and longer to run.
> * The check to wait when there are no files to process will never pass
> because the size of the data structure will never be 0. This could lead to
> tight looped checks and some utilization issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
