[
https://issues.apache.org/jira/browse/JCLOUDS-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-701:
--------------------------------
Component/s: jclouds-compute
> createNodesInGroup fails with IllegalStateException if we hit CPU quota on GCE
> ------------------------------------------------------------------------------
>
> Key: JCLOUDS-701
> URL: https://issues.apache.org/jira/browse/JCLOUDS-701
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-compute, jclouds-labs-google
> Affects Versions: 1.8.0
> Reporter: Sunil Shah
>
> When we call createNodesInGroup for a Google Compute Engine project that has
> come up against its CPU quota, we see this exception:
> {code}
> 2014-09-02 15:55:03,640 ERROR [pool-3-thread-4] j.compute
> [SLF4JLogger.java:93] createNodesInGroup(nothing), completed: 0/2, errors: 1,
> rate: 6586ms/op
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> operation failed. Http Error Code: 403 HttpError: FORBIDDEN
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-17.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-17.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-17.0.jar:na]
> at
> org.jclouds.concurrent.FutureIterables$1.run(FutureIterables.java:123)
> ~[jclouds-core-1.8.0.jar:1.8.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
> Caused by: java.lang.IllegalStateException: operation failed. Http Error
> Code: 403 HttpError: FORBIDDEN
> at
> org.jclouds.googlecomputeengine.compute.GoogleComputeEngineServiceAdapter.waitOperationDone(GoogleComputeEngineServiceAdapter.java:434)
> ~[google-compute-engine-1.8.1-20140814.210916-6.jar:1.8.1-SNAPSHOT]
> at
> org.jclouds.googlecomputeengine.compute.GoogleComputeEngineServiceAdapter.createNodeWithGroupEncodedIntoName(GoogleComputeEngineServiceAdapter.java:185)
> ~[google-compute-engine-1.8.1-20140814.210916-6.jar:1.8.1-SNAPSHOT]
> at
> org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.createNodeWithGroupEncodedIntoName(AdaptingComputeServiceStrategies.java:195)
> ~[jclouds-compute-1.8.0.jar:1.8.0]
> at
> org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:79)
> ~[jclouds-compute-1.8.0.jar:1.8.0]
> at
> org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:63)
> ~[jclouds-compute-1.8.0.jar:1.8.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
> at java.lang.Thread.getStackTrace(Thread.java:1589) [na:1.7.0_60]
> at
> org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.createNodeInGroupWithNameAndTemplate(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:167)
> ~[jclouds-compute-1.8.0.jar:1.8.0]
> at
> org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:122)
> ~[jclouds-compute-1.8.0.jar:1.8.0]
> at
> org.jclouds.googlecomputeengine.compute.strategy.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:127)
> ~[google-compute-engine-1.8.1-20140814.210916-6.jar:1.8.1-SNAPSHOT]
> at
> org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:210)
> ~[jclouds-compute-1.8.0.jar:1.8.0]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_60]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.7.0_60]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_60]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_60]
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> ~[guice-3.0.jar:na]
> at com.sun.proxy.$Proxy124.createNodesInGroup(Unknown Source) ~[na:na]
> at
> mesosphere.poseidon.providers.Google.createNodesFromTemplate(Google.scala:58)
> ~[classes/:na]
> {code}
> This is inconsistent with the error we get when the network quota is
> exceeded, which is:
> {code}
> 2014-08-07 22:01:37,109 WARN [ApiWorker-2] m.p.p.Google [Provider.scala:142]
> development-24-88d79664.dcc.io // Retrying after:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalStateException: Could not create network, operation
> failedOperation{kind=compute#operation, id=3290207288552397268,
> name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860,
> selfLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/operations/operation-1407448895523-500113b7508b8-d2ab879d-4773f860,
> name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860,
> targetLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/networks/jclouds-development-24,
> status=DONE, [email protected], progress=100, insertTime=Thu Aug 07
> 22:01:35 UTC 2014, startTime=Thu Aug 07 22:01:35 UTC 2014, endTime=Thu Aug 07
> 22:01:36 UTC 2014, httpError={statusCode=403, message=FORBIDDEN, headers={}},
> operationType=insert, errors=[Error{code=QUOTA_EXCEEDED, message=Quota
> 'NETWORKS' exceeded. Limit: 5.0}]}
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4812)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.apply(LocalCache.java:4830)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.jclouds.googlecomputeengine.compute.strategy.CreateNodesWithGroupEncodedIntoNameThenAddToSet.getOrCreateNetwork(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:137)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.jclouds.googlecomputeengine.compute.strategy.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:122)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:215)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_55]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.7.0_55]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_55]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55]
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at com.sun.proxy.$Proxy124.createNodesInGroup(Unknown Source) ~[na:na]
> at
> mesosphere.poseidon.providers.Google$$anonfun$1$$anonfun$apply$1.apply(Google.scala:58)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> mesosphere.poseidon.providers.Google$$anonfun$1$$anonfun$apply$1.apply(Google.scala:55)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> [poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> [poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
> Caused by: java.lang.IllegalStateException: Could not create network,
> operation failedOperation{kind=compute#operation, id=3290207288552397268,
> name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860,
> selfLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/operations/operation-1407448895523-500113b7508b8-d2ab879d-4773f860,
> name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860,
> targetLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/networks/jclouds-development-24,
> status=DONE, [email protected], progress=100, insertTime=Thu Aug 07
> 22:01:35 UTC 2014, startTime=Thu Aug 07 22:01:35 UTC 2014, endTime=Thu Aug 07
> 22:01:36 UTC 2014, httpError={statusCode=403, message=FORBIDDEN, headers={}},
> operationType=insert, errors=[Error{code=QUOTA_EXCEEDED, message=Quota
> 'NETWORKS' exceeded. Limit: 5.0}]}
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:150)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.jclouds.googlecomputeengine.functions.CreateNetworkIfNeeded.apply(CreateNetworkIfNeeded.java:95)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.jclouds.googlecomputeengine.functions.CreateNetworkIfNeeded.apply(CreateNetworkIfNeeded.java:46)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.jclouds.googlecomputeengine.compute.loaders.FindNetworkOrCreate.load(FindNetworkOrCreate.java:59)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.jclouds.googlecomputeengine.compute.loaders.FindNetworkOrCreate.load(FindNetworkOrCreate.java:36)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193)
> ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
> ... 21 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
