[
https://issues.apache.org/jira/browse/HADOOP-10794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054657#comment-14054657
]
Steve Loughran commented on HADOOP-10794:
-----------------------------------------
I'm not sure if we can/should be doing much more than recommend NTP and setting
time zones properly (if a host has it's TZ env var wrong then it can appear to
be hours out, even if it is in sync). NTP sets it host-wide, calculates local
clock drift and compensates for it, and has good availability.
FWIW Ant's diagnostics method not only prints out TZ info, it looks at the temp
dir and compares its clock with the local time, as the situation "NAS filestore
in different time to host" can cause chaos with dependency logic
{code}
-------------------------------------------
Temp dir
-------------------------------------------
Temp dir is /var/folders/57/xyts0qt105z1f1k0twk6rd8m0000gq/T/
Temp dir is writeable
Temp dir alignment with system clock is -819 ms
-------------------------------------------
Locale information
-------------------------------------------
Timezone Greenwich Mean Time offset=3600000
{code}
For Hadoop, the problem will be worst in VMs, as their clocks will be jerky and
may even go backwards if a VM is moved to another physical host. NTP isn't so
good there as its a a use case it doesn't expect.
Ignoring that, the real risk is "ops think a machine is syncing its clock with
NTP but isn't". I've seen this happen.
We may want hadoop to help catch that by looking at clocks across a cluster and
warning if some is "significantly" off. This could be done with the RM catching
the times of hosts when they report in -and flagging when one is beyond a
configured threshold. Then it'll be left to that ops team to fix it however
they do it in the cluster.
> A hadoop cluster needs clock synchronization
> --------------------------------------------
>
> Key: HADOOP-10794
> URL: https://issues.apache.org/jira/browse/HADOOP-10794
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Zhijie Shen
>
> As a distributed system, a hadoop cluster wants the clock on all the
> participating hosts synchronized. Otherwise, some problems might happen. For
> example, in YARN-2251, due to the clock on the host for the task container
> falls behind that on the host of the AM container, the computed elapsed time
> (the diff between the timestamps produced on two hosts) becomes negative.
> In YARN-2251, we tried to mask the negative elapsed time. However, we should
> seek for a decent long term solution, such as providing mechanism to do and
> check clock synchronization.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
