Junping Du created MAPREDUCE-6680:
-------------------------------------

             Summary: JHS UserLogDir scan algorithm sometime could skip 
directory with update in CloudFS (Azure FileSystem, S3, etc.)
                 Key: MAPREDUCE-6680
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: jobhistoryserver
            Reporter: Junping Du
            Assignee: Junping Du


In our cluster based on a Cloud FileSystem, we notice JHS sometimes could skip 
directory with .jhist file in scanning.
The behavior is like:
First round scan, doesn't found .jhist file:
{noformat}
16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a directory 
with 6 files in it.
16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
...
{noformat}

Then, we see "Scan not needed of ..." for the same directory every 3 minutes 
until application failed as timeout.

>From our analysis, we found the root cause is: most of Cloud File System 
>(Azure, S3, etc.) is truncating file/directory modification time to seconds 
>instead of milliseconds - which could due to limit of http protocol (from 
>discussion at: https://forums.aws.amazon.com/thread.jspa?messageID=476615). 

So if the time sequence is happen to be: latest non .jhist file modification on 
directory happens at T1, directory scanning happens at T2, .jhist file added to 
directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3 after 
truncating to seconds, this issue could appear.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to