Hello, I'm using chef with the knife-cloudstack plugin to create and configure cloud instances from the command line. The knife-cloudstack plugin just issues API commands to the server. Here is the command I'm entering, with the error output:
nixbox:chef-repo nico$ bundle exec knife cs server create testinstance -N testinstance -r 'role[emn_dev]' -E dev –no-public-ip WARNING: Could not load IOV methods. Check your GSSAPI C library for an update WARNING: Could not load AEAD methods. Check your GSSAPI C library for an update Waiting for Server to be created....... Allocate ip address, create forwarding rules Error 533: { "associateipaddressresponse": { "errorcode": 533, "cserrorcode": 9999, "errortext": "Insufficient address capacity" } } URL: https://cloud.evolvingmedia.org/client/api?apiKey=<removed>&command=associateIpAddress&networkId=2ddfc21a-b37d-4bd5-8bde-efc19858fa4c&response=json&zoneId=7b9b0354-486f-4d94-957d-8f6b6e26f56f&signature=J4AWM3RyM0VRMknMlaMlf9fSbAA%3D I've researched this and requested help on IRC for a few days I a row now, but haven't had any luck finding help. These are the log entries that appear to be related to the issue (from apilog.log): 2014-05-21 14:15:28,308 INFO [cloud.api.ApiServer] (catalina-exec-23:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&available=true&command=listZones&response=json&signature=2DeOmGxaud81DUPFAxG24o9DRp8%3D 200 { "listzonesresponse" : { "count":1 ,"zone" : [ {"id":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","name":"288 Wall Street","dns1":"10.32.0.1","dns2":"","internaldns1":"10.32.0.1","internaldns2":"","domain":"cloud.evolvingmedia.org","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","domainname":"ROOT","networktype":"Basic","securitygroupsenabled":true,"allocationstate":"Enabled","zonetoken":"8d701628-c7ec-3fac-8d2c-7b1261c095d1","dhcpprovider":"VirtualRouter","localstorageenabled":false} ] } } 2014-05-21 14:15:28,508 INFO [cloud.api.ApiServer] (catalina-exec-12:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=listTemplates&response=json&templateFilter=executable&zoneid=7b9b0354-486f-4d94-957d-8f6b6e26f56f&signature=5UOQAo0Et6x%2FWl1X2qJtAzuAcAM%3D 200 { "listtemplatesresponse" : { "count":6 ,"template" : [ {"id":"120936ab-8fab-465d-b3a7-b7dcb3670021","name":"CentOS 5.6(64-bit) no GUI (XenServer)","displaytext":"CentOS 5.6(64-bit) no GUI (XenServer)","ispublic":true,"created":"2012-06-14T18:20:35-0400","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":false,"crossZones":true,"ostypeid":"a72c1198-db72-4862-a656-2e5c701438e9","ostypename":"CentOS 5.3 (64-bit)","account":"system","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":21474836480,"templatetype":"BUILTIN","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":true,"checksum":"60d0c09446c8b5eff142440b69856d83","tags":[],"sshkeyenabled":false}, {"id":"b5b9778a-d1e4-46c2-8c16-d29d248f539b","name":"Ubuntu 12.04 LTS x86_64 Min PV CHEF OLD","displaytext":"updated 1/9/14","ispublic":true,"created":"2014-01-09T11:27:00-0500","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":true,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"sourcetemplateid":"6a554220-884a-4cf1-8a36-1cae35c21ba7","tags":[],"sshkeyenabled":false}, {"id":"22da3730-3dc7-4b3c-af21-8eb3745126e2","name":"Ubuntu 12.04 LTS x86_64 Min PV CHEF","displaytext":"Updated 1/9/14","ispublic":true,"created":"2014-01-09T14:22:26-0500","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":true,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"sourcetemplateid":"b5b9778a-d1e4-46c2-8c16-d29d248f539b","tags":[],"sshkeyenabled":false}, {"id":"e4170e40-ade8-415a-aed3-f9f01e6b49f4","name":"mysql-rescue","displaytext":"mysql rescued 10/30/12","ispublic":false,"created":"2012-10-30T03:08:15-0400","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":false,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"checksum":"3fbdc18bd1b83b5bfb0ecbe66d162137","tags":[],"sshkeyenabled":false}, {"id":"2327a4a4-3ce4-40f4-a559-634d8457f744","name":"Samba","displaytext":"samba","ispublic":false,"created":"2012-11-20T16:50:58-0500","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":false,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"checksum":"b3e6f7e33678baf6b882ae642b2ae71c","sourcetemplateid":"29131a3f-ed54-4468-a174-ba055fab4973","tags":[],"sshkeyenabled":false}, {"id":"55ac39b6-33ba-4eb2-b0b1-14401cce99da","name":"Ubuntu 12.04 LTS x86_64 Kerb/LDAP 288","displaytext":"Updated 12/11/12","ispublic":true,"created":"2012-12-11T17:39:58-0500","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":true,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"checksum":"bcf808b13cdb73dc3ef1432020476e94","tags":[],"sshkeyenabled":false} ] } } 2014-05-21 14:15:28,651 INFO [cloud.api.ApiServer] (catalina-exec-14:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=listVirtualMachines&name=testinstance4&response=json&signature=wkwid%2Bmtvhl197kO89nslLkYaOE%3D 200 { "listvirtualmachinesresponse" : { } } 2014-05-21 14:15:28,782 INFO [cloud.api.ApiServer] (catalina-exec-1:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=listServiceOfferings&response=json&signature=2dDqNqI1LinViCc63keQ1ZQQmzQ%3D 200 { "listserviceofferingsresponse" : { "count":16 ,"serviceoffering" : [ {"id":"81f6576e-e39d-464c-92f7-babeb7b051d5","name":"2x1Ghz 4GB","displaytext":"2x1Ghz 4GB","cpunumber":2,"cpuspeed":1000,"memory":4096,"created":"2013-01-25T13:22:11-0500","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"d1dfaf07-4ae7-4eb3-b351-c0d1cc9e958a","name":"Medium Instance","displaytext":"Medium Instance","cpunumber":1,"cpuspeed":1000,"memory":1024,"created":"2012-06-14T18:06:55-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"fc02f066-a884-4179-902d-942b4cd10f09","name":"1x1GHz 512MB","displaytext":"1x1GHz 512MB","cpunumber":1,"cpuspeed":1000,"memory":512,"created":"2012-06-27T21:44:35-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"94f52e31-be3a-4141-be4e-81bfa5f4c0f5","name":"00_default","displaytext":"default","cpunumber":1,"cpuspeed":1000,"memory":512,"created":"2013-05-07T18:20:45-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"d78f37be-02ec-425f-b942-8a8a9d53aa87","name":"1x300Mhz 128MB","displaytext":"Ubuntu Min Spec","cpunumber":1,"cpuspeed":300,"memory":128,"created":"2012-08-24T10:31:44-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"d4ec4613-7d9a-4dfb-8d27-473c18ad6287","name":"2x2Ghz 4GB","displaytext":"2x2Ghz 4GB","cpunumber":2,"cpuspeed":1000,"memory":4000,"created":"2012-06-18T11:41:53-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"1488ee53-223d-406b-accd-8857837b31ed","name":"2x1.5Ghz 4GB","displaytext":"2x1.5Ghz 4GB","cpunumber":2,"cpuspeed":1500,"memory":4096,"created":"2013-01-22T16:16:48-0500","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"8eec48ab-80f7-4c34-af74-cdab29e69684","name":"Small Instance","displaytext":"Small Instance","cpunumber":1,"cpuspeed":500,"memory":512,"created":"2012-06-14T18:06:55-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"7b56a2c3-df5e-404d-8d91-9675a3e7fcbe","name":"1x1.5Ghz 1G","displaytext":"1x1.5Ghz 1G","cpunumber":1,"cpuspeed":1500,"memory":1000,"created":"2012-06-18T16:13:12-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"9b1761fa-b8e5-409f-9de0-a1c14080d3e0","name":"1x1GHz 2.5G","displaytext":"coloxen01 ratio","cpunumber":1,"cpuspeed":1000,"memory":2560,"created":"2013-03-06T11:07:45-0500","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"82ec93ef-c975-4c22-b113-91b39532f7a6","name":"1x1Ghz 1GB","displaytext":"1x1Ghz 1GB","cpunumber":1,"cpuspeed":1000,"memory":1000,"created":"2012-06-18T11:38:35-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"afa50d17-770a-47fb-92c0-550df1fd522e","name":"High CPU","displaytext":"4x3Ghz 4GB","cpunumber":4,"cpuspeed":3000,"memory":4000,"created":"2012-07-24T14:54:08-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"bb12589a-c647-48d6-a473-e30ff108c2f8","name":"2x2GHz 8G","displaytext":"2x2GHz 8G","cpunumber":2,"cpuspeed":2000,"memory":8192,"created":"2012-12-20T12:02:34-0500","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"5d5e774e-b66f-4c50-86e7-72f8153b792c","name":"4x2GHz 10GB","displaytext":"4x2GHz 10GB","cpunumber":4,"cpuspeed":2000,"memory":10240,"created":"2013-08-14T17:11:51-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"996c8f6d-65c7-4ed4-af28-3df3a25b160d","name":"1x1Ghz 2GB","displaytext":"1x1Ghz 2GB","cpunumber":1,"cpuspeed":1000,"memory":2000,"created":"2012-06-18T11:49:18-0400","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false}, {"id":"df0b9997-e9c1-4613-86fd-d83f4d7758d5","name":"1x.5GHz 512MB","displaytext":"1x.5GHz 512MB","cpunumber":1,"cpuspeed":500,"memory":512,"created":"2012-12-13T15:19:42-0500","storagetype":"shared","offerha":false,"limitcpuuse":false,"issystem":false,"defaultuse":false} ] } } 2014-05-21 14:15:28,918 INFO [cloud.api.ApiServer] (catalina-exec-5:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&available=true&command=listZones&response=json&signature=2DeOmGxaud81DUPFAxG24o9DRp8%3D 200 { "listzonesresponse" : { "count":1 ,"zone" : [ {"id":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","name":"288 Wall Street","dns1":"10.32.0.1","dns2":"","internaldns1":"10.32.0.1","internaldns2":"","domain":"cloud.evolvingmedia.org","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","domainname":"ROOT","networktype":"Basic","securitygroupsenabled":true,"allocationstate":"Enabled","zonetoken":"8d701628-c7ec-3fac-8d2c-7b1261c095d1","dhcpprovider":"VirtualRouter","localstorageenabled":false} ] } } 2014-05-21 14:15:29,089 INFO [cloud.api.ApiServer] (catalina-exec-24:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=listTemplates&response=json&templateFilter=executable&zoneid=7b9b0354-486f-4d94-957d-8f6b6e26f56f&signature=5UOQAo0Et6x%2FWl1X2qJtAzuAcAM%3D 200 { "listtemplatesresponse" : { "count":6 ,"template" : [ {"id":"120936ab-8fab-465d-b3a7-b7dcb3670021","name":"CentOS 5.6(64-bit) no GUI (XenServer)","displaytext":"CentOS 5.6(64-bit) no GUI (XenServer)","ispublic":true,"created":"2012-06-14T18:20:35-0400","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":false,"crossZones":true,"ostypeid":"a72c1198-db72-4862-a656-2e5c701438e9","ostypename":"CentOS 5.3 (64-bit)","account":"system","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":21474836480,"templatetype":"BUILTIN","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":true,"checksum":"60d0c09446c8b5eff142440b69856d83","tags":[],"sshkeyenabled":false}, {"id":"b5b9778a-d1e4-46c2-8c16-d29d248f539b","name":"Ubuntu 12.04 LTS x86_64 Min PV CHEF OLD","displaytext":"updated 1/9/14","ispublic":true,"created":"2014-01-09T11:27:00-0500","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":true,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"sourcetemplateid":"6a554220-884a-4cf1-8a36-1cae35c21ba7","tags":[],"sshkeyenabled":false}, {"id":"22da3730-3dc7-4b3c-af21-8eb3745126e2","name":"Ubuntu 12.04 LTS x86_64 Min PV CHEF","displaytext":"Updated 1/9/14","ispublic":true,"created":"2014-01-09T14:22:26-0500","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":true,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"sourcetemplateid":"b5b9778a-d1e4-46c2-8c16-d29d248f539b","tags":[],"sshkeyenabled":false}, {"id":"e4170e40-ade8-415a-aed3-f9f01e6b49f4","name":"mysql-rescue","displaytext":"mysql rescued 10/30/12","ispublic":false,"created":"2012-10-30T03:08:15-0400","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":false,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"checksum":"3fbdc18bd1b83b5bfb0ecbe66d162137","tags":[],"sshkeyenabled":false}, {"id":"2327a4a4-3ce4-40f4-a559-634d8457f744","name":"Samba","displaytext":"samba","ispublic":false,"created":"2012-11-20T16:50:58-0500","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":false,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"checksum":"b3e6f7e33678baf6b882ae642b2ae71c","sourcetemplateid":"29131a3f-ed54-4468-a174-ba055fab4973","tags":[],"sshkeyenabled":false}, {"id":"55ac39b6-33ba-4eb2-b0b1-14401cce99da","name":"Ubuntu 12.04 LTS x86_64 Kerb/LDAP 288","displaytext":"Updated 12/11/12","ispublic":true,"created":"2012-12-11T17:39:58-0500","isready":true,"passwordenabled":false,"format":"VHD","isfeatured":true,"crossZones":false,"ostypeid":"a3611a04-8238-41cb-9f51-9ef361a4c265","ostypename":"Other PV (64-bit)","account":"admin","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","status":"Download Complete","size":5368709120,"templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","isextractable":false,"checksum":"bcf808b13cdb73dc3ef1432020476e94","tags":[],"sshkeyenabled":false} ] } } 2014-05-21 14:15:29,229 INFO [cloud.api.ApiServer] (catalina-exec-18:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&available=true&command=listZones&response=json&signature=2DeOmGxaud81DUPFAxG24o9DRp8%3D 200 { "listzonesresponse" : { "count":1 ,"zone" : [ {"id":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","name":"288 Wall Street","dns1":"10.32.0.1","dns2":"","internaldns1":"10.32.0.1","internaldns2":"","domain":"cloud.evolvingmedia.org","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","domainname":"ROOT","networktype":"Basic","securitygroupsenabled":true,"allocationstate":"Enabled","zonetoken":"8d701628-c7ec-3fac-8d2c-7b1261c095d1","dhcpprovider":"VirtualRouter","localstorageenabled":false} ] } } 2014-05-21 14:15:30,174 INFO [cloud.api.ApiServer] (catalina-exec-25:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=deployVirtualMachine&displayname=testinstance4&name=testinstance4&response=json&serviceOfferingId=d78f37be-02ec-425f-b942-8a8a9d53aa87&templateId=22da3730-3dc7-4b3c-af21-8eb3745126e2&zoneId=7b9b0354-486f-4d94-957d-8f6b6e26f56f&signature=J9uVtsKCI06vEPMawGsjywBIAtQ%3D 200 { "deployvirtualmachineresponse" : {"id":"1b0ce3ca-2988-4fa8-8dfb-f83f4bd0050d","jobid":"b45d6542-d3ad-4ed5-ba82-dc12c4140fa8"} } 2014-05-21 14:15:30,701 INFO [cloud.api.ApiServer] (catalina-exec-3:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=queryAsyncJobResult&jobId=b45d6542-d3ad-4ed5-ba82-dc12c4140fa8&response=json&signature=wSeYn%2FN0EkdSZhs50HiNINWnT7o%3D 200 { "queryasyncjobresultresponse" : {"accountid":"ad8bbb18-556c-42df-be10-8f66a92c26e3","userid":"14728534-0632-402c-8416-6acf811190d1","cmd":"org.apache.cloudstack.api.command.user.vm.DeployVMCmd","jobstatus":0,"jobprocstatus":0,"jobresultcode":0,"jobinstancetype":"VirtualMachine","jobinstanceid":"1b0ce3ca-2988-4fa8-8dfb-f83f4bd0050d","created":"2014-05-21T14:15:30-0400","jobid":"b45d6542-d3ad-4ed5-ba82-dc12c4140fa8"} } 2014-05-21 14:15:35,984 INFO [cloud.api.ApiServer] (catalina-exec-21:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=queryAsyncJobResult&jobId=b45d6542-d3ad-4ed5-ba82-dc12c4140fa8&response=json&signature=wSeYn%2FN0EkdSZhs50HiNINWnT7o%3D 200 { "queryasyncjobresultresponse" : {"accountid":"ad8bbb18-556c-42df-be10-8f66a92c26e3","userid":"14728534-0632-402c-8416-6acf811190d1","cmd":"org.apache.cloudstack.api.command.user.vm.DeployVMCmd","jobstatus":0,"jobprocstatus":0,"jobresultcode":0,"jobinstancetype":"VirtualMachine","jobinstanceid":"1b0ce3ca-2988-4fa8-8dfb-f83f4bd0050d","created":"2014-05-21T14:15:30-0400","jobid":"b45d6542-d3ad-4ed5-ba82-dc12c4140fa8"} } 2014-05-21 14:15:41,324 INFO [cloud.api.ApiServer] (catalina-exec-20:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=queryAsyncJobResult&jobId=b45d6542-d3ad-4ed5-ba82-dc12c4140fa8&response=json&signature=wSeYn%2FN0EkdSZhs50HiNINWnT7o%3D 200 { "queryasyncjobresultresponse" : {"accountid":"ad8bbb18-556c-42df-be10-8f66a92c26e3","userid":"14728534-0632-402c-8416-6acf811190d1","cmd":"org.apache.cloudstack.api.command.user.vm.DeployVMCmd","jobstatus":0,"jobprocstatus":0,"jobresultcode":0,"jobinstancetype":"VirtualMachine","jobinstanceid":"1b0ce3ca-2988-4fa8-8dfb-f83f4bd0050d","created":"2014-05-21T14:15:30-0400","jobid":"b45d6542-d3ad-4ed5-ba82-dc12c4140fa8"} } 2014-05-21 14:15:46,627 INFO [cloud.api.ApiServer] (catalina-exec-13:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=queryAsyncJobResult&jobId=b45d6542-d3ad-4ed5-ba82-dc12c4140fa8&response=json&signature=wSeYn%2FN0EkdSZhs50HiNINWnT7o%3D 200 { "queryasyncjobresultresponse" : {"accountid":"ad8bbb18-556c-42df-be10-8f66a92c26e3","userid":"14728534-0632-402c-8416-6acf811190d1","cmd":"org.apache.cloudstack.api.command.user.vm.DeployVMCmd","jobstatus":0,"jobprocstatus":0,"jobresultcode":0,"jobinstancetype":"VirtualMachine","jobinstanceid":"1b0ce3ca-2988-4fa8-8dfb-f83f4bd0050d","created":"2014-05-21T14:15:30-0400","jobid":"b45d6542-d3ad-4ed5-ba82-dc12c4140fa8"} } 2014-05-21 14:15:51,969 INFO [cloud.api.ApiServer] (catalina-exec-17:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=queryAsyncJobResult&jobId=b45d6542-d3ad-4ed5-ba82-dc12c4140fa8&response=json&signature=wSeYn%2FN0EkdSZhs50HiNINWnT7o%3D 200 { "queryasyncjobresultresponse" : {"accountid":"ad8bbb18-556c-42df-be10-8f66a92c26e3","userid":"14728534-0632-402c-8416-6acf811190d1","cmd":"org.apache.cloudstack.api.command.user.vm.DeployVMCmd","jobstatus":0,"jobprocstatus":0,"jobresultcode":0,"jobinstancetype":"VirtualMachine","jobinstanceid":"1b0ce3ca-2988-4fa8-8dfb-f83f4bd0050d","created":"2014-05-21T14:15:30-0400","jobid":"b45d6542-d3ad-4ed5-ba82-dc12c4140fa8"} } 2014-05-21 14:15:57,248 INFO [cloud.api.ApiServer] (catalina-exec-22:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=queryAsyncJobResult&jobId=b45d6542-d3ad-4ed5-ba82-dc12c4140fa8&response=json&signature=wSeYn%2FN0EkdSZhs50HiNINWnT7o%3D 200 { "queryasyncjobresultresponse" : {"accountid":"ad8bbb18-556c-42df-be10-8f66a92c26e3","userid":"14728534-0632-402c-8416-6acf811190d1","cmd":"org.apache.cloudstack.api.command.user.vm.DeployVMCmd","jobstatus":0,"jobprocstatus":0,"jobresultcode":0,"jobinstancetype":"VirtualMachine","jobinstanceid":"1b0ce3ca-2988-4fa8-8dfb-f83f4bd0050d","created":"2014-05-21T14:15:30-0400","jobid":"b45d6542-d3ad-4ed5-ba82-dc12c4140fa8"} } 2014-05-21 14:16:02,381 INFO [cloud.api.ApiServer] (catalina-exec-15:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=queryAsyncJobResult&jobId=b45d6542-d3ad-4ed5-ba82-dc12c4140fa8&response=json&signature=wSeYn%2FN0EkdSZhs50HiNINWnT7o%3D 200 { "queryasyncjobresultresponse" : {"accountid":"ad8bbb18-556c-42df-be10-8f66a92c26e3","userid":"14728534-0632-402c-8416-6acf811190d1","cmd":"org.apache.cloudstack.api.command.user.vm.DeployVMCmd","jobstatus":1,"jobprocstatus":0,"jobresultcode":0,"jobresulttype":"object","jobresult":{"virtualmachine":{"id":"1b0ce3ca-2988-4fa8-8dfb-f83f4bd0050d","name":"testinstance4","displayname":"testinstance4","account":"admin","domainid":"30914512-f84a-4d37-832e-b7acb0c09191","domain":"ROOT","created":"2014-05-21T14:15:29-0400","state":"Running","haenable":false,"zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","hostid":"c5cafcff-741b-45de-8185-2ad1b76647ce","hostname":"xenserver","templateid":"22da3730-3dc7-4b3c-af21-8eb3745126e2","templatename":"Ubuntu 12.04 LTS x86_64 Min PV CHEF","templatedisplaytext":"Updated 1/9/14","passwordenabled":false,"serviceofferingid":"d78f37be-02ec-425f-b942-8a8a9d53aa87","serviceofferingname":"1x300Mhz 128MB","cpunumber":1,"cpuspeed":300,"memory":128,"guestosid":"a3611a04-8238-41cb-9f51-9ef361a4c265","rootdeviceid":0,"rootdevicetype":"ROOT","securitygroup":[],"nic":[{"id":"ae7a9eb2-66bb-4b1d-9dd6-4c0f401d8628","networkid":"2ddfc21a-b37d-4bd5-8bde-efc19858fa4c","networkname":"guestNetworkForBasicZone","netmask":"255.255.248.0","gateway":"10.32.0.1","ipaddress":"10.32.2.42","isolationuri":"ec2://untagged","broadcasturi":"vlan://untagged","traffictype":"Guest","type":"Shared","isdefault":true,"macaddress":"06:6b:fc:00:01:60"}],"hypervisor":"XenServer","instancename":"i-2-496-VM","tags":[],"jobid":"b45d6542-d3ad-4ed5-ba82-dc12c4140fa8","jobstatus":0}},"created":"2014-05-21T14:15:30-0400","jobid":"b45d6542-d3ad-4ed5-ba82-dc12c4140fa8"} } 2014-05-21 14:16:02,527 INFO [cloud.api.ApiServer] (catalina-exec-2:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=listNetworks&isDefault=true&response=json&zoneid=7b9b0354-486f-4d94-957d-8f6b6e26f56f&signature=LiXa5KwhMM406lGfhGWvOd0IPxA%3D 200 { "listnetworksresponse" : { "count":1 ,"network" : [ {"id":"2ddfc21a-b37d-4bd5-8bde-efc19858fa4c","name":"guestNetworkForBasicZone","displaytext":"guestNetworkForBasicZone","broadcastdomaintype":"Vlan","traffictype":"Guest","gateway":"10.32.0.1","zoneid":"7b9b0354-486f-4d94-957d-8f6b6e26f56f","zonename":"288 Wall Street","networkofferingid":"14d01132-c63f-428d-a154-b7b4f1c4894d","networkofferingname":"DefaultSharedNetworkOffering","networkofferingdisplaytext":"Offering for Shared networks","networkofferingavailability":"Optional","issystem":false,"state":"Setup","related":"2ddfc21a-b37d-4bd5-8bde-efc19858fa4c","dns1":"10.32.0.1","dns2":"","type":"Shared","acltype":"Domain","subdomainaccess":true,"domainid":"30914512-f84a-4d37-832e-b7acb0c09191","domain":"ROOT","service":[{"name":"Dns","capability":[{"name":"AllowDnsSuffixModification","value":"true","canchooseservicecapability":false}]},{"name":"UserData"},{"name":"Dhcp"}],"networkdomain":"cloud.evolvingmedia.org","physicalnetworkid":"08a7c2af-8628-4127-8bf0-d33c31ecf1d3","restartrequired":false,"specifyipranges":true,"canusefordeploy":true,"ispersistent":false,"tags":[]} ] } } 2014-05-21 14:16:02,865 INFO [cloud.api.ApiServer] (catalina-exec-4:null) (userId=2 accountId=2 sessionId=null) 127.0.0.1 -- GET apiKey=<removed>&command=associateIpAddress&networkId=2ddfc21a-b37d-4bd5-8bde-efc19858fa4c&response=json&zoneId=7b9b0354-486f-4d94-957d-8f6b6e26f56f&signature=J4AWM3RyM0VRMknMlaMlf9fSbAA%3D 533 Insufficient address capacity The issue here is that the IP address pool for this network should not be exceeded, the pool is roughly 200 IP's, with only about 25 hosts actually having IP reservations within the network. I have read that in some cases, expunged instances don't release their IP's, and I have looked at the vm_instances table, and I do see well over 300 records, most of which are in the expunged state. It seems likely that the IP pool needs to be cleaned up manually from the database, but I'm not sure how to confirm that is the case – and if I DO, I don't really know which sql statements to run on which tables to get this handled. Thanks! Nick Abbruzzese Developer Operations Evolving Media Network 288 Wall Street Kingston, NY 12401 Phone: (845) 328 3220 x119 Email: n...@evolvingmedia.net<mailto:n...@evolvingmedia.net>