My alarm has nothing to do with the word SEVERE, it’s more due to the fact that my thread dies. I’m just using the blobstore clojure api, I’m not familiar with the async vs sync denotation in coordination with that API.
<http://www.datastax.com/> PETER HALLIDAY OpsCenter | Sr Software Engineer | 607.329.6905 | pe...@datastax.com <mailto:pe...@datastax.com> <https://twitter.com/phalliday> <https://www.linkedin.com/profile/view?id=7030852> > On Dec 5, 2014, at 11:12 AM, Andrew Gaul <g...@apache.org > <mailto:g...@apache.org>> wrote: > > [Moving from the dev to the user mailing list.] > > The SEVERE log message is not as alarming as you might suspect; JDK > logger maps error level logs to severe level. It appears that you are > using the deprecated async jclouds interface instead of the sync > interface; are you consuming the error response via Future.get? > Otherwise, can you minimize a test case so we can diagnose this further? > > On Fri, Dec 05, 2014 at 10:41:48AM -0600, Peter Halliday wrote: >> I’m using JClouds BlobStore in an application that’s using it to send files >> to S3. It does this in batches like after a snapshot is run on a database. >> And then all of the files are transferred to S3. We were testing error >> states. And we were testing changing a User’s Access credentials from >> inactive to active and back. When we go from active to inactive credentials >> during this transfer to S3. Our thread gets killed. The last log entry is >> from JClouds. Our code is literally catching all errors, and so nothing is >> being thrown at that time. Our application is in clojure. You can see in >> the log, that we upload successfully fine, and then after we switch to >> invalid we “correct” get errors that get returned to our application. >> However, then we hit the SEVERE log line, and then our thread is dead and >> never comes back :-( >> >> >> >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,398 Sending >> message >> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Data.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,496 Uploading file: >> snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-CompressionInfo.db >> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,498 Throttle value requested >> is out of range, using default. >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,695 Sending >> message >> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-CompressionInfo.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,797 Uploading file: >> snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-Index.db >> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,809 Throttle value requested >> is out of range, using default. >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,105 Sending >> message >> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,204 Sending >> message >> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Statistics.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,305 Sending >> message >> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,406 Sending >> message >> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> ERROR [async-dispatch-31] 2014-12-05 16:29:15,503 Got exception HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending >> opsagent.backups.entities.SSTable@48fee98c to >> :380a2db2ac2042c5bd43043d8ae53b6f >> org.jclouds.rest.AuthorizationException: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122) >> at >> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94) >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89) >> at >> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) >> at com.sun.proxy.$Proxy48.objectExists(Unknown Source) >> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173) >> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238) >> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43) >> at >> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123) >> at >> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944) >> at >> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953) >> at >> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133) >> at clojure.lang.AFn.run(AFn.java:24) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: org.jclouds.http.HttpResponseException: request: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db> >> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63) >> ... 25 more >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,504 Sending >> message '["172.31.10.206",{"details":{"error":"HEAD >> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com >> <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Summary.db >> HTTP\/1.1 -> HTTP\/1.1 403 >> Forbidden","filename":"OpsCenter-rollups300-jb-96-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> ERROR [async-dispatch-31] 2014-12-05 16:29:15,600 Got exception HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending >> opsagent.backups.entities.SSTable@91afeb42 to >> :380a2db2ac2042c5bd43043d8ae53b6f >> org.jclouds.rest.AuthorizationException: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122) >> at >> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94) >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89) >> at >> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) >> at com.sun.proxy.$Proxy48.objectExists(Unknown Source) >> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173) >> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238) >> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43) >> at >> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123) >> at >> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944) >> at >> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953) >> at >> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133) >> at clojure.lang.AFn.run(AFn.java:24) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: org.jclouds.http.HttpResponseException: request: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db> >> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63) >> ... 25 more >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,601 Sending >> message '["172.31.10.206",{"details":{"error":"HEAD >> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com >> <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Filter.db >> HTTP\/1.1 -> HTTP\/1.1 403 >> Forbidden","filename":"OpsCenter-rollups300-jb-96-Filter.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> ERROR [async-dispatch-31] 2014-12-05 16:29:15,696 Got exception HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending >> opsagent.backups.entities.SSTable@a4c93 to :380a2db2ac2042c5bd43043d8ae53b6f >> org.jclouds.rest.AuthorizationException: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122) >> at >> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94) >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89) >> at >> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) >> at com.sun.proxy.$Proxy48.objectExists(Unknown Source) >> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173) >> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238) >> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43) >> at >> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123) >> at >> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944) >> at >> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953) >> at >> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133) >> at clojure.lang.AFn.run(AFn.java:24) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: org.jclouds.http.HttpResponseException: request: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db> >> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63) >> ... 25 more >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,697 Sending >> message '["172.31.10.206",{"details":{"error":"HEAD >> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com >> <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-CompressionInfo.db >> HTTP\/1.1 -> HTTP\/1.1 403 >> Forbidden","filename":"OpsCenter-pdps-jb-114-CompressionInfo.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> ERROR [async-dispatch-31] 2014-12-05 16:29:15,792 Got exception HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending >> opsagent.backups.entities.SSTable@44e5d97 to >> :380a2db2ac2042c5bd43043d8ae53b6f >> org.jclouds.rest.AuthorizationException: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122) >> at >> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94) >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89) >> at >> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) >> at com.sun.proxy.$Proxy48.objectExists(Unknown Source) >> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173) >> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238) >> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43) >> at >> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123) >> at >> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944) >> at >> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953) >> at >> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133) >> at clojure.lang.AFn.run(AFn.java:24) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: org.jclouds.http.HttpResponseException: request: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db> >> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63) >> ... 25 more >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,793 Sending >> message '["172.31.10.206",{"details":{"error":"HEAD >> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com >> <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Statistics.db >> HTTP\/1.1 -> HTTP\/1.1 403 >> Forbidden","filename":"OpsCenter-pdps-jb-114-Statistics.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> ERROR [async-dispatch-31] 2014-12-05 16:29:15,888 Got exception HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending >> opsagent.backups.entities.SSTable@8ca4473b to >> :380a2db2ac2042c5bd43043d8ae53b6f >> org.jclouds.rest.AuthorizationException: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db> >> HTTP/1.1 -> HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122) >> at >> org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94) >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89) >> at >> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135) >> at >> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94) >> at >> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) >> at >> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) >> at com.sun.proxy.$Proxy48.objectExists(Unknown Source) >> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173) >> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238) >> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43) >> at >> opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123) >> at >> opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44) >> at >> opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940) >> at >> clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944) >> at >> clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953) >> at >> clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133) >> at clojure.lang.AFn.run(AFn.java:24) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: org.jclouds.http.HttpResponseException: request: HEAD >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db> >> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden >> at >> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63) >> ... 25 more >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,890 Sending >> message '["172.31.10.206",{"details":{"error":"HEAD >> https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com >> <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Data.db >> HTTP\/1.1 -> HTTP\/1.1 403 >> Forbidden","filename":"OpsCenter-pdps-jb-114-Data.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Starting real-time collection >> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Collecting real-time data >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,034 Sending >> message '["172.31.10.206",{"os-load":0.07}]' to channel '/node-details'. >> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,298 Sending >> message >> '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"schema","state":"running","details":{"keyspace":"OpsCenter","filename":"OpsCenter\/schema.json"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' >> to channel '/snapshot/status'. >> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,402 Uploading file: >> snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db >> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,404 Throttle value requested >> is out of range, using default. >> ERROR [async-dispatch-31] 2014-12-05 16:29:16,510 Dec 05, 2014 4:29:16 PM >> org.jclouds.logging.jdk.JDKLogger logError >> SEVERE: Cannot retry after server error, command is not replayable: >> [method=org.jclouds.aws.s3.AWSS3AsyncClient.public abstract >> com.google.common.util.concurrent.ListenableFuture >> org.jclouds.s3.S3AsyncClient.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[phalliday-opscenter-backups-4, >> >> [metadata=[key=snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db, >> bucket=null, uri=null, eTag=null, cacheControl=null, >> contentMetadata=[contentDisposition=null, contentEncoding=null, >> contentLanguage=null, contentLength=107, contentMD5=null, >> contentType=application/octet-stream, expires=null], lastModified=null, >> owner=null, storageClass=STANDARD, userMetadata={}]], >> [Lorg.jclouds.s3.options.PutObjectOptions;@4e0ecbfe], request=PUT >> https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db >> >> <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db> >> HTTP/1.1] >> >> DEBUG [node-details-1] 2014-12-05 16:29:21,034 Starting real-time collection >> DEBUG [node-details-1] 2014-12-05 16:29:21,035 Collecting real-time data >> DEBUG [node-details-1] 2014-12-05 16:29:21,038 Send requested, but nothing >> new to send >> >> >> >> >> >> <http://www.datastax.com/ <http://www.datastax.com/>> >> PETER HALLIDAY >> OpsCenter | Sr Software Engineer | 607.329.6905 | pe...@datastax.com >> <mailto:pe...@datastax.com> >> >> <https://twitter.com/phalliday <https://twitter.com/phalliday>> >> <https://www.linkedin.com/profile/view?id=7030852 >> <https://www.linkedin.com/profile/view?id=7030852>> > > -- > Andrew Gaul > http://gaul.org/ <http://gaul.org/>
