[Moving from the dev to the user mailing list.]
The SEVERE log message is not as alarming as you might suspect; JDK
logger maps error level logs to severe level. It appears that you are
using the deprecated async jclouds interface instead of the sync
interface; are you consuming the error response via Future.get?
Otherwise, can you minimize a test case so we can diagnose this further?
On Fri, Dec 05, 2014 at 10:41:48AM -0600, Peter Halliday wrote:
> I’m using JClouds BlobStore in an application that’s using it to send files
> to S3. It does this in batches like after a snapshot is run on a database.
> And then all of the files are transferred to S3. We were testing error
> states. And we were testing changing a User’s Access credentials from
> inactive to active and back. When we go from active to inactive credentials
> during this transfer to S3. Our thread gets killed. The last log entry is
> from JClouds. Our code is literally catching all errors, and so nothing is
> being thrown at that time. Our application is in clojure. You can see in
> the log, that we upload successfully fine, and then after we switch to
> invalid we “correct” get errors that get returned to our application.
> However, then we hit the SEVERE log line, and then our thread is dead and
> never comes back :-(
>
>
>
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,398 Sending message
> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Data.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,496 Uploading file:
> snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-CompressionInfo.db
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,498 Throttle value requested is
> out of range, using default.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,695 Sending message
> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-CompressionInfo.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,797 Uploading file:
> snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-Index.db
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,809 Throttle value requested is
> out of range, using default.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,105 Sending message
> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,204 Sending message
> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Statistics.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,305 Sending message
> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,406 Sending message
> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,503 Got exception HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
> opsagent.backups.entities.SSTable@48fee98c to
> :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at
> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at
> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at
> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at
> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at
> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,504 Sending message
> '["172.31.10.206",{"details":{"error":"HEAD
> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Summary.db
> HTTP\/1.1 -> HTTP\/1.1 403
> Forbidden","filename":"OpsCenter-rollups300-jb-96-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,600 Got exception HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
> opsagent.backups.entities.SSTable@91afeb42 to
> :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at
> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at
> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at
> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at
> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at
> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,601 Sending message
> '["172.31.10.206",{"details":{"error":"HEAD
> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Filter.db
> HTTP\/1.1 -> HTTP\/1.1 403
> Forbidden","filename":"OpsCenter-rollups300-jb-96-Filter.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,696 Got exception HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
> opsagent.backups.entities.SSTable@a4c93 to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at
> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at
> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at
> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at
> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at
> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,697 Sending message
> '["172.31.10.206",{"details":{"error":"HEAD
> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-CompressionInfo.db
> HTTP\/1.1 -> HTTP\/1.1 403
> Forbidden","filename":"OpsCenter-pdps-jb-114-CompressionInfo.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,792 Got exception HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
> opsagent.backups.entities.SSTable@44e5d97 to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at
> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at
> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at
> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at
> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at
> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,793 Sending message
> '["172.31.10.206",{"details":{"error":"HEAD
> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Statistics.db
> HTTP\/1.1 -> HTTP\/1.1 403
> Forbidden","filename":"OpsCenter-pdps-jb-114-Statistics.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,888 Got exception HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
> opsagent.backups.entities.SSTable@8ca4473b to
> :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
> HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at
> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at
> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at
> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at
> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at
> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at
> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at
> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,890 Sending message
> '["172.31.10.206",{"details":{"error":"HEAD
> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Data.db
> HTTP\/1.1 -> HTTP\/1.1 403
> Forbidden","filename":"OpsCenter-pdps-jb-114-Data.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Starting real-time collection
> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Collecting real-time data
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,034 Sending message
> '["172.31.10.206",{"os-load":0.07}]' to channel '/node-details'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,298 Sending message
> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"schema","state":"running","details":{"keyspace":"OpsCenter","filename":"OpsCenter\/schema.json"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
> to channel '/snapshot/status'.
> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,402 Uploading file:
> snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,404 Throttle value requested is
> out of range, using default.
> ERROR [async-dispatch-31] 2014-12-05 16:29:16,510 Dec 05, 2014 4:29:16 PM
> org.jclouds.logging.jdk.JDKLogger logError
> SEVERE: Cannot retry after server error, command is not replayable:
> [method=org.jclouds.aws.s3.AWSS3AsyncClient.public abstract
> com.google.common.util.concurrent.ListenableFuture
> org.jclouds.s3.S3AsyncClient.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[phalliday-opscenter-backups-4,
>
> [metadata=[key=snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db,
> bucket=null, uri=null, eTag=null, cacheControl=null,
> contentMetadata=[contentDisposition=null, contentEncoding=null,
> contentLanguage=null, contentLength=107, contentMD5=null,
> contentType=application/octet-stream, expires=null], lastModified=null,
> owner=null, storageClass=STANDARD, userMetadata={}]],
> [Lorg.jclouds.s3.options.PutObjectOptions;@4e0ecbfe], request=PUT
> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
> HTTP/1.1]
>
> DEBUG [node-details-1] 2014-12-05 16:29:21,034 Starting real-time collection
> DEBUG [node-details-1] 2014-12-05 16:29:21,035 Collecting real-time data
> DEBUG [node-details-1] 2014-12-05 16:29:21,038 Send requested, but nothing
> new to send
>
>
>
>
>
> <http://www.datastax.com/>
> PETER HALLIDAY
> OpsCenter | Sr Software Engineer | 607.329.6905 | [email protected]
>
> <https://twitter.com/phalliday>
> <https://www.linkedin.com/profile/view?id=7030852>
--
Andrew Gaul
http://gaul.org/
