On Tuesday, May 14, 2013, John Burwell wrote:
> Chip,
>
> After some further discussion about this issue on IRC, Alex and I
> determined that system VM clock drift issue not only breaks S3, but has
> other significant impacts that merit it being a blocker for 4.1 (e.g.
> timestamps of files written by the SSVM being incorrect, log file
> correlation difficulties, sensitivity of other storage protocols to time
> sync, etc). Based on this conversation, I have opened
> CLOUDSTACK-2492<https://issues.apache.org/jira/browse/CLOUDSTACK-2492> to
> capture the issue and track resolution.
>
> Thanks,
> -John
>
Well shoot. Issuing a new system VM is going to require a bunch of testing
if it's not done as a hand rolled edit of the existing system VMs.
Who can take this?
>
> On May 14, 2013, at 6:09 PM, John Burwell
> <jburw...@basho.com<javascript:_e({}, 'cvml', 'jburw...@basho.com');>>
> wrote:
>
> Chip,
>
> The source of the problem appears to be clock drift between the SSVM and
> S3 per following stack trace:
>
> 2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils]
> (agentRequest-Handler-3:) Putting directory
> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in
> S3 bucket jsb-cloudstack-templates.
> 2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils]
> (agentRequest-Handler-3:) Creating S3 client with configuration: [protocol:
> https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000]
> 2013-05-14 06:51:55,403 DEBUG
> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
> Determining key using account id 1 and template id 5
> 2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils]
> (agentRequest-Handler-3:) Putting file
> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties
> into bucket jsb-cloudstack-templates with key
> template/tmpl/1/5/template.properties.
> 2013-05-14 06:51:55,578 ERROR
> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
> Failed to upload template id 5
> Status Code: 403, AWS Service: Amazon S3, AWS Request ID:
> 970A274E132A9ACB, AWS Error Code: RequestTimeTooSkewed, AWS Error Message:
> The difference between the request time and the current time is too large.,
> S3 Extended Request ID:
> 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
> at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
> at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
> at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
> at
> com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
> at
> com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
> at com.cloud.agent.Agent.processRequest(Agent.java:525)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> It does not appear that the System VM image has NTP nor does it seem like
> a valid assumption that an SSVM would have the connectivity to reach an NTP
> server. Therefore, do we have some other means to sync the clock on the
> SSVM (e.g. with the host through hypervisor extensions)? In my previous
> tests, the SSVM seemed to be syncing its time to the host which addressed
> the problem.
>
> Thanks,
> -John
>
> On May 14, 2013, at 4:58 PM, John Burwell
> <jburw...@basho.com<javascript:_e({}, 'cvml', 'jburw...@basho.com');>>
> wrote:
>
> Chip,
>
> I am looking into the issue now. There is a failure when the S3 upload
> template command is issued. I working to determine whether or not the
> cause is environmental or code.
>
> Thanks,
> -John
>
> On May 14, 2013, at 4:56 PM, Chip Childers
> <chip.child...@sungard.com<javascript:_e({}, 'cvml',
> 'chip.child...@sungard.com');>>
> wrote:
>
> Hi all,
>
> We have a clear bug list (blockers and critical) for 4.1.0. I'm going
> to cut a new release candidate tonight.
>
> If there are *any* outstanding issues known, now's the time to raise
> them.
>
> (I'm specifically looking for an ACK from jburwell here, since he
> mentioned a possible S3 feature issue. John, if you can please give me
> a heads up on where you stand in the next few hours, I'd appreciate it.)
>
> -chip
>
>
>
>
>
