[jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian He updated MAPREDUCE-6680:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.7.3
2.8.0
Status: Resolved (was: Patch Available)
Committed to trunk, branch-2, branch-2.8, branch-2.7, thanks Junping !
> JHS UserLogDir scan algorithm sometime could skip directory with update in
> CloudFS (Azure FileSystem, S3, etc.)
> ---
>
> Key: MAPREDUCE-6680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
> Labels: Azure, S3
> Fix For: 2.8.0, 2.7.3
>
> Attachments: MAPREDUCE-6680-v2.patch, MAPREDUCE-6680-v3.patch,
> MAPREDUCE-6680.patch
>
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could
> skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a
> directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes
> until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System
> (Azure FS, S3, etc.) is truncating file/directory modification time to
> seconds instead of milliseconds - which could due to limit of http protocol
> (from discussion at:
> https://forums.aws.amazon.com/thread.jspa?messageID=476615).
> So if the time sequence is happen to be: latest non .jhist file modification
> on directory happens at T1, directory scanning happens at T2, .jhist file
> added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3
> after truncating to seconds, this issue could appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6680:
--
Labels: Azure S3 (was: )
> JHS UserLogDir scan algorithm sometime could skip directory with update in
> CloudFS (Azure FileSystem, S3, etc.)
> ---
>
> Key: MAPREDUCE-6680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
> Labels: Azure, S3
> Attachments: MAPREDUCE-6680-v2.patch, MAPREDUCE-6680-v3.patch,
> MAPREDUCE-6680.patch
>
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could
> skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a
> directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes
> until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System
> (Azure FS, S3, etc.) is truncating file/directory modification time to
> seconds instead of milliseconds - which could due to limit of http protocol
> (from discussion at:
> https://forums.aws.amazon.com/thread.jspa?messageID=476615).
> So if the time sequence is happen to be: latest non .jhist file modification
> on directory happens at T1, directory scanning happens at T2, .jhist file
> added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3
> after truncating to seconds, this issue could appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6680:
--
Attachment: MAPREDUCE-6680-v3.patch
> JHS UserLogDir scan algorithm sometime could skip directory with update in
> CloudFS (Azure FileSystem, S3, etc.)
> ---
>
> Key: MAPREDUCE-6680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: MAPREDUCE-6680-v2.patch, MAPREDUCE-6680-v3.patch,
> MAPREDUCE-6680.patch
>
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could
> skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a
> directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes
> until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System
> (Azure FS, S3, etc.) is truncating file/directory modification time to
> seconds instead of milliseconds - which could due to limit of http protocol
> (from discussion at:
> https://forums.aws.amazon.com/thread.jspa?messageID=476615).
> So if the time sequence is happen to be: latest non .jhist file modification
> on directory happens at T1, directory scanning happens at T2, .jhist file
> added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3
> after truncating to seconds, this issue could appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6680:
--
Attachment: (was: MAPREDUCE-6680-v3.patch)
> JHS UserLogDir scan algorithm sometime could skip directory with update in
> CloudFS (Azure FileSystem, S3, etc.)
> ---
>
> Key: MAPREDUCE-6680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: MAPREDUCE-6680-v2.patch, MAPREDUCE-6680.patch
>
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could
> skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a
> directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes
> until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System
> (Azure FS, S3, etc.) is truncating file/directory modification time to
> seconds instead of milliseconds - which could due to limit of http protocol
> (from discussion at:
> https://forums.aws.amazon.com/thread.jspa?messageID=476615).
> So if the time sequence is happen to be: latest non .jhist file modification
> on directory happens at T1, directory scanning happens at T2, .jhist file
> added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3
> after truncating to seconds, this issue could appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6680:
--
Attachment: MAPREDUCE-6680-v3.patch
> JHS UserLogDir scan algorithm sometime could skip directory with update in
> CloudFS (Azure FileSystem, S3, etc.)
> ---
>
> Key: MAPREDUCE-6680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: MAPREDUCE-6680-v2.patch, MAPREDUCE-6680-v3.patch,
> MAPREDUCE-6680.patch
>
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could
> skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a
> directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes
> until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System
> (Azure FS, S3, etc.) is truncating file/directory modification time to
> seconds instead of milliseconds - which could due to limit of http protocol
> (from discussion at:
> https://forums.aws.amazon.com/thread.jspa?messageID=476615).
> So if the time sequence is happen to be: latest non .jhist file modification
> on directory happens at T1, directory scanning happens at T2, .jhist file
> added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3
> after truncating to seconds, this issue could appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6680:
--
Attachment: MAPREDUCE-6680-v2.patch
v2 patch fix checkstyle issue.
> JHS UserLogDir scan algorithm sometime could skip directory with update in
> CloudFS (Azure FileSystem, S3, etc.)
> ---
>
> Key: MAPREDUCE-6680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: MAPREDUCE-6680-v2.patch, MAPREDUCE-6680.patch
>
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could
> skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a
> directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes
> until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System
> (Azure FS, S3, etc.) is truncating file/directory modification time to
> seconds instead of milliseconds - which could due to limit of http protocol
> (from discussion at:
> https://forums.aws.amazon.com/thread.jspa?messageID=476615).
> So if the time sequence is happen to be: latest non .jhist file modification
> on directory happens at T1, directory scanning happens at T2, .jhist file
> added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3
> after truncating to seconds, this issue could appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6680:
--
Status: Patch Available (was: Open)
> JHS UserLogDir scan algorithm sometime could skip directory with update in
> CloudFS (Azure FileSystem, S3, etc.)
> ---
>
> Key: MAPREDUCE-6680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: MAPREDUCE-6680.patch
>
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could
> skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a
> directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes
> until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System
> (Azure FS, S3, etc.) is truncating file/directory modification time to
> seconds instead of milliseconds - which could due to limit of http protocol
> (from discussion at:
> https://forums.aws.amazon.com/thread.jspa?messageID=476615).
> So if the time sequence is happen to be: latest non .jhist file modification
> on directory happens at T1, directory scanning happens at T2, .jhist file
> added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3
> after truncating to seconds, this issue could appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6680:
--
Attachment: MAPREDUCE-6680.patch
Put a patch with proper fix. The unit test could be very hard given the issue
appears to be platform related and intermittently. I think the fix is
straight-forward enough.
> JHS UserLogDir scan algorithm sometime could skip directory with update in
> CloudFS (Azure FileSystem, S3, etc.)
> ---
>
> Key: MAPREDUCE-6680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: MAPREDUCE-6680.patch
>
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could
> skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a
> directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes
> until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System
> (Azure FS, S3, etc.) is truncating file/directory modification time to
> seconds instead of milliseconds - which could due to limit of http protocol
> (from discussion at:
> https://forums.aws.amazon.com/thread.jspa?messageID=476615).
> So if the time sequence is happen to be: latest non .jhist file modification
> on directory happens at T1, directory scanning happens at T2, .jhist file
> added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3
> after truncating to seconds, this issue could appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6680:
--
Description:
In our cluster based on a Cloud FileSystem, we notice JHS sometimes could skip
directory with .jhist file in scanning.
The behavior is like:
First round scan, doesn't found .jhist file:
{noformat}
16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a directory
with 6 files in it.
16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
...
{noformat}
Then, we see "Scan not needed of ..." for the same directory every 3 minutes
until application failed as timeout.
>From our analysis, we found the root cause is: most of Cloud File System
>(Azure FS, S3, etc.) is truncating file/directory modification time to seconds
>instead of milliseconds - which could due to limit of http protocol (from
>discussion at: https://forums.aws.amazon.com/thread.jspa?messageID=476615).
So if the time sequence is happen to be: latest non .jhist file modification on
directory happens at T1, directory scanning happens at T2, .jhist file added to
directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3 after
truncating to seconds, this issue could appear.
was:
In our cluster based on a Cloud FileSystem, we notice JHS sometimes could skip
directory with .jhist file in scanning.
The behavior is like:
First round scan, doesn't found .jhist file:
{noformat}
16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a directory
with 6 files in it.
16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
...
{noformat}
Then, we see "Scan not needed of ..." for the same directory every 3 minutes
until application failed as timeout.
>From our analysis, we found the root cause is: most of Cloud File System
>(Azure, S3, etc.) is truncating file/directory modification time to seconds
>instead of milliseconds - which could due to limit of http protocol (from
>discussion at: https://forums.aws.amazon.com/thread.jspa?messageID=476615).
So if the time sequence is happen to be: latest non .jhist file modification on
directory happens at T1, directory scanning happens at T2, .jhist file added to
directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3 after
truncating to seconds, this issue could appear.
> JHS UserLogDir scan algorithm sometime could skip directory with update in
> CloudFS (Azure FileSystem, S3, etc.)
> ---
>
> Key: MAPREDUCE-6680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could
> skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a
> directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes
> until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System
> (Azure FS, S3, etc.) is truncating file/directory modification time to
> seconds instead of milliseconds - which could due to limit of http protocol
> (from discussion at:
> https://forums.aws.amazon.com/thread.jspa?messageID=476615).
> So if the time sequence is happen to be: latest non .jhist file modification
> on directory happens at T1, directory scanning happens at T2, .jhist file
> added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3
> after truncating to seconds, this issue could appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
