John Burwell created CLOUDSTACK-2492:
----------------------------------------
Summary: System VMs Clock Drift
Key: CLOUDSTACK-2492
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2492
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: ISO
Affects Versions: 4.1.0
Environment: devcloud/Xen
Reporter: John Burwell
Priority: Blocker
Testing of S3-backed Secondary Storage has revealed that the SSVM (and likely
all other system VMs) have no provision for clock synchronization (e.g. NTP to
dom0 for Xen, vmware-tools for VMWare, etc). In particular, the S3 protocol is
sensitive to drift between clients and S3. As an example, the following is the
stack trace caused by clock drift S3:
2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:)
Putting directory
/mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in S3
bucket jsb-cloudstack-templates.
2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:)
Creating S3 client with configuration: [protocol: https, connectionTimeOut:
50000, maxErrorRetry: 3, socketTimeout: 50000]
2013-05-14 06:51:55,403 DEBUG [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-3:) Determining key using account id 1 and template id 5
2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:)
Putting file
/mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties
into bucket jsb-cloudstack-templates with key
template/tmpl/1/5/template.properties.
2013-05-14 06:51:55,578 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-3:) Failed to upload template id 5
Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 970A274E132A9ACB, AWS
Error Code: RequestTimeTooSkewed, AWS Error Message: The difference between the
request time and the current time is too large., S3 Extended Request ID:
9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
at
com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
at
com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
In addition to impacting S3, this clock drift also makes log correlation
between the management server and system VMs very difficult, if not,
impossible. Finally, there are suspicions that the clock drift could also
impact operation of console proxy and virtual router VMs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
