Problem with NAT
Hello community :-) I have two CS installations, one in version 4.10 and one 4.11. In both cases, the infrastructure is configured in the same way, we have an advanced network. In 4.11 everything works fine, while in 4.10, static NAT and port forwarding from public addresses do not work properly in the VPC network. The only difference I found was that in 4.11 in Virtual Router, the network looks like this: eth0 - link local eth1 - vr public / nat and port fw ip's (logically this is ok) eth2 - vpc In 4.10 instead: eth0 - link local eth1 - vr public ip eth2 - vpc / public ip with nat or port fw enabled (logically this is not ok) The question is, is VR in 4.10 configured correctly? How this diagnose? The VPC network has a default_allow policy, in both cases we can get outside. I can not find any error in the configuration, in the previously existing configuration 4.10 everything worked correctly (we test different possibilities). Regards, Piotr
Re: Unable to schedule async job
/etc/cloudstack/management/log4j-cloud.xml more info: https://image.slidesharecdn.com/cloudstacktroubleshooting-lily-150506063617-conversion-gate02/95/cloud-stack-troubleshooting-12-638.jpg?cb=1430894327 cheers On Thu, 10 Jan 2019 at 19:17, Ivan X Yue wrote: > Dag, > > That's good idea. How can I update log to debug / trace mode? > > Thanks > Ivan > > > > From: Dag Sonstebo > To: "users@cloudstack.apache.org" > Date: 2019/01/10 11:40 AM > Subject:Re: Unable to schedule async job > > > > Hi Ivan, > > Probably a good idea to bump your logging up to debug or trace – these > show the SQL queries being prepared in the logs. > > Regards, > Dag Sonstebo > Cloud Architect > ShapeBlue > > > From: Ivan X Yue > Reply-To: "users@cloudstack.apache.org" > Date: Thursday, 10 January 2019 at 16:18 > To: "users@cloudstack.apache.org" > Subject: Re: Unable to schedule async job > > Hi, Dag, > > Thank for the reply. > > Below is my async_job table. Today, I try to delete some VM. > Interestingly, I can delete some VM and some are failed withthis error: > > Caused by: > com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: > > Duplicate entry '387' for key 'PRIMARY' > > But in the async_job table, I don't see id=387 there. > > I have attached part of the Management Server log wrt the delete VM > action. > > One minor correction. I am actually using cloudstack 4.11.1 instead of > 4.9.2. > > MariaDB [cloud]> select id, job_cmd from async_job order by id; > +-+-+ > | id | job_cmd | > +-+-+ > | 275 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | > | 276 | NULL| > | 281 | NULL| > | 287 | NULL| > | 288 | NULL| > | 292 | NULL| > | 305 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | > | 307 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | > | 309 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | > | 313 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | > | 314 | NULL| > | 315 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | > | 316 | com.cloud.vm.VmWorkStart| > | 317 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | > | 319 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | > | 321 | NULL| > | 322 | com.cloud.vm.VmWorkStop | > | 323 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | > | 327 | com.cloud.vm.VmWorkStart| > | 328 | com.cloud.vm.VmWorkStart| > | 329 | NULL| > | 330 | com.cloud.vm.VmWorkStop | > | 332 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | > | 334 | com.cloud.vm.VmWorkStop | > | 335 | com.cloud.vm.VmWorkStart| > | 336 | com.cloud.vm.VmWorkStart| > | 338 | org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin | > | 340 | com.cloud.vm.VmWorkStop | > | 341 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | > | 342 | com.cloud.vm.VmWorkStop | > | 343 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| > | 344 | com.cloud.vm.VmWorkStart| > | 345 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | > | 346 | com.cloud.vm.VmWorkStop | > | 347 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | > | 348 | com.cloud.vm.VmWorkStop | > | 349 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| > | 350 | com.cloud.vm.VmWorkStart| > | 351 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | > | 352 | com.cloud.vm.VmWorkStop | > | 354 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | > | 355 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | > | 356 | com.cloud.vm.VmWorkStop
Re: Unable to schedule async job
Dag, That's good idea. How can I update log to debug / trace mode? Thanks Ivan From: Dag Sonstebo To: "users@cloudstack.apache.org" Date: 2019/01/10 11:40 AM Subject:Re: Unable to schedule async job Hi Ivan, Probably a good idea to bump your logging up to debug or trace – these show the SQL queries being prepared in the logs. Regards, Dag Sonstebo Cloud Architect ShapeBlue From: Ivan X Yue Reply-To: "users@cloudstack.apache.org" Date: Thursday, 10 January 2019 at 16:18 To: "users@cloudstack.apache.org" Subject: Re: Unable to schedule async job Hi, Dag, Thank for the reply. Below is my async_job table. Today, I try to delete some VM. Interestingly, I can delete some VM and some are failed withthis error: Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '387' for key 'PRIMARY' But in the async_job table, I don't see id=387 there. I have attached part of the Management Server log wrt the delete VM action. One minor correction. I am actually using cloudstack 4.11.1 instead of 4.9.2. MariaDB [cloud]> select id, job_cmd from async_job order by id; +-+-+ | id | job_cmd | +-+-+ | 275 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 276 | NULL| | 281 | NULL| | 287 | NULL| | 288 | NULL| | 292 | NULL| | 305 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 307 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 309 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 313 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | | 314 | NULL| | 315 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | | 316 | com.cloud.vm.VmWorkStart| | 317 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 319 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 321 | NULL| | 322 | com.cloud.vm.VmWorkStop | | 323 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 327 | com.cloud.vm.VmWorkStart| | 328 | com.cloud.vm.VmWorkStart| | 329 | NULL| | 330 | com.cloud.vm.VmWorkStop | | 332 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 334 | com.cloud.vm.VmWorkStop | | 335 | com.cloud.vm.VmWorkStart| | 336 | com.cloud.vm.VmWorkStart| | 338 | org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin | | 340 | com.cloud.vm.VmWorkStop | | 341 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 342 | com.cloud.vm.VmWorkStop | | 343 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| | 344 | com.cloud.vm.VmWorkStart| | 345 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 346 | com.cloud.vm.VmWorkStop | | 347 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 348 | com.cloud.vm.VmWorkStop | | 349 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| | 350 | com.cloud.vm.VmWorkStart| | 351 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 352 | com.cloud.vm.VmWorkStop | | 354 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 355 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 356 | com.cloud.vm.VmWorkStop | | 357 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots| | 358 | com.cloud.vm.VmWorkStop | | 359 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 360 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 361 | com.cloud.vm.VmWorkStop | | 362 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots
Re: KVM Agent Error: Incorrect details for private Nic
Hi Daniel, Yes makes sense and this is your simplest setup. Just remembered I updated this blog post a couple of months back - it should see you right https://www.shapeblue.com/networking-kvm-for-cloudstack-2018-revisit-for-centos7-and-ubuntu-18-04/ Regards, Dag Sonstebo Cloud Architect ShapeBlue On 10/01/2019, 16:37, "Daniel Farrar" wrote: Thank you Dag, this looks very likely to be the case. I was able to find a scenario yesterday that brought the service up by setting eno1.8 type to Ethernet, disabling eno1.8, restarting the agent service, then re-enabling eno1.8 as type Vlan. My intended design was to create a 2 NIC bond (for now just single NIC eno1 for troubleshooting), create subinterfaces (ex eno1.8 for vlan8), and assign each subinterface to a cloudstack service. I will try letting Cloudstack take care of all vlans and remove the subinterface, pointing cloudbr0 directly to eno1 or bond. Daniel Farrar -Original Message- From: Dag Sonstebo [mailto:dag.sonst...@shapeblue.com] Sent: Wednesday, January 9, 2019 10:38 AM To: users@cloudstack.apache.org Subject: Re: KVM Agent Error: Incorrect details for private Nic Hi Daniel, My initial guess without deep diving into this is your NIC naming. The CloudStack agent looks for the interface (bond / NIC / team) which supports cloudbr0 - and I think maybe in your case it can't find it due to the dotted naming "eno1.8" when it probably expects some conventional like "eno16777984". Is there a reason you've named it like this? This would normally point to a VLAN setup (whereas cloudbr0 needs to be trunked in an advanced zone). If not can you rename it? Code for this is somewhere around here I think: https://github.com/apache/cloudstack/blob/c565db2cf2ab5fdbd2fba65409928dfa7c5f2d25/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java#L1312 Developers - please correct me if I'm wrong. Regards, Dag Sonstebo Cloud Architect ShapeBlue On 09/01/2019, 14:08, "Daniel Farrar" wrote: Hello! Wanted to see if anyone else has ran into this issue or if you can point me in the right direction... I have 2 KVM CloudStack-agents version 4.11.2 running Centos7 that are not able to start the cloudstack-agent service if the agent.propreties file specifies the variable "private.network.device=cloudbr0". This is automatically added to the config file when the agent connects to the server, if the private network is commented out the service is able to start up but any system VMs are unable to connect to the network. This environment was working with my original 2 KVM agents, but after deleting the hosts and rebuilding with identical network config I've ran into this issue. I have logs and configs below, any ideas what is wrong? This is a dev proof of concept environment, so I can rebuild if needed. Thank you for your time! 2019-01-07 15:36:10,257 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2019-01-07 15:36:10,259 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.11.2.0 2019-01-07 15:36:10,260 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2019-01-07 15:36:10,271 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage 2019-01-07 15:36:10,272 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm 2019-01-07 15:36:10,282 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml 2019-01-07 15:36:10,293 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection 2019-01-07 15:36:10,384 INFO [cloud.agent.Agent] (main:null) (logid:) id is 15 2019-01-07 15:36:10,387 WARN [cloud.resource.ServerResourceBase] (main:null) (logid:) Incorrect details for private Nic during initialization of ServerResourceBase 2019-01-07 15:36:10,387 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: Unable to configure LibvirtComputingResource This occurs on version 4.11.1 and 4.11.2, and I've seen other threads with this issue on earlier versions that reported downgrading the agent fixed this issue. NIC CONFIGURATION: ifcfg-cloudbr0 DEVICE=cloudbr0 TYPE=Bridge ONBOOT=yes BOOTPROTO=none IPV6INIT=no IPV6_AUTOCONF=no DELAY=5 STP=yes Ifcfg-eno1.8 HWADDR=XX:XX:XX:XX:XX:XX DEVICE=eno1.8 ONBOOT=yes HOTPLUG=no
Re: Unable to schedule async job
Hi Ivan, Probably a good idea to bump your logging up to debug or trace – these show the SQL queries being prepared in the logs. Regards, Dag Sonstebo Cloud Architect ShapeBlue From: Ivan X Yue Reply-To: "users@cloudstack.apache.org" Date: Thursday, 10 January 2019 at 16:18 To: "users@cloudstack.apache.org" Subject: Re: Unable to schedule async job Hi, Dag, Thank for the reply. Below is my async_job table. Today, I try to delete some VM. Interestingly, I can delete some VM and some are failed withthis error: Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '387' for key 'PRIMARY' But in the async_job table, I don't see id=387 there. I have attached part of the Management Server log wrt the delete VM action. One minor correction. I am actually using cloudstack 4.11.1 instead of 4.9.2. MariaDB [cloud]> select id, job_cmd from async_job order by id; +-+-+ | id | job_cmd | +-+-+ | 275 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 276 | NULL| | 281 | NULL| | 287 | NULL| | 288 | NULL| | 292 | NULL| | 305 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 307 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 309 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 313 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | | 314 | NULL| | 315 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | | 316 | com.cloud.vm.VmWorkStart| | 317 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 319 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 321 | NULL| | 322 | com.cloud.vm.VmWorkStop | | 323 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 327 | com.cloud.vm.VmWorkStart| | 328 | com.cloud.vm.VmWorkStart| | 329 | NULL| | 330 | com.cloud.vm.VmWorkStop | | 332 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 334 | com.cloud.vm.VmWorkStop | | 335 | com.cloud.vm.VmWorkStart| | 336 | com.cloud.vm.VmWorkStart| | 338 | org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin | | 340 | com.cloud.vm.VmWorkStop | | 341 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 342 | com.cloud.vm.VmWorkStop | | 343 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| | 344 | com.cloud.vm.VmWorkStart| | 345 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 346 | com.cloud.vm.VmWorkStop | | 347 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 348 | com.cloud.vm.VmWorkStop | | 349 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| | 350 | com.cloud.vm.VmWorkStart| | 351 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 352 | com.cloud.vm.VmWorkStop | | 354 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 355 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 356 | com.cloud.vm.VmWorkStop | | 357 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots| | 358 | com.cloud.vm.VmWorkStop | | 359 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 360 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 361 | com.cloud.vm.VmWorkStop | | 362 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots| | 363 | com.cloud.vm.VmWorkStop | | 364 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 365 | com.cloud.vm.VmWorkStop
RE: KVM Agent Error: Incorrect details for private Nic
Thank you Dag, this looks very likely to be the case. I was able to find a scenario yesterday that brought the service up by setting eno1.8 type to Ethernet, disabling eno1.8, restarting the agent service, then re-enabling eno1.8 as type Vlan. My intended design was to create a 2 NIC bond (for now just single NIC eno1 for troubleshooting), create subinterfaces (ex eno1.8 for vlan8), and assign each subinterface to a cloudstack service. I will try letting Cloudstack take care of all vlans and remove the subinterface, pointing cloudbr0 directly to eno1 or bond. Daniel Farrar -Original Message- From: Dag Sonstebo [mailto:dag.sonst...@shapeblue.com] Sent: Wednesday, January 9, 2019 10:38 AM To: users@cloudstack.apache.org Subject: Re: KVM Agent Error: Incorrect details for private Nic Hi Daniel, My initial guess without deep diving into this is your NIC naming. The CloudStack agent looks for the interface (bond / NIC / team) which supports cloudbr0 - and I think maybe in your case it can't find it due to the dotted naming "eno1.8" when it probably expects some conventional like "eno16777984". Is there a reason you've named it like this? This would normally point to a VLAN setup (whereas cloudbr0 needs to be trunked in an advanced zone). If not can you rename it? Code for this is somewhere around here I think: https://github.com/apache/cloudstack/blob/c565db2cf2ab5fdbd2fba65409928dfa7c5f2d25/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java#L1312 Developers - please correct me if I'm wrong. Regards, Dag Sonstebo Cloud Architect ShapeBlue On 09/01/2019, 14:08, "Daniel Farrar" wrote: Hello! Wanted to see if anyone else has ran into this issue or if you can point me in the right direction... I have 2 KVM CloudStack-agents version 4.11.2 running Centos7 that are not able to start the cloudstack-agent service if the agent.propreties file specifies the variable "private.network.device=cloudbr0". This is automatically added to the config file when the agent connects to the server, if the private network is commented out the service is able to start up but any system VMs are unable to connect to the network. This environment was working with my original 2 KVM agents, but after deleting the hosts and rebuilding with identical network config I've ran into this issue. I have logs and configs below, any ideas what is wrong? This is a dev proof of concept environment, so I can rebuild if needed. Thank you for your time! 2019-01-07 15:36:10,257 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2019-01-07 15:36:10,259 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.11.2.0 2019-01-07 15:36:10,260 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2019-01-07 15:36:10,271 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage 2019-01-07 15:36:10,272 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm 2019-01-07 15:36:10,282 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml 2019-01-07 15:36:10,293 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection 2019-01-07 15:36:10,384 INFO [cloud.agent.Agent] (main:null) (logid:) id is 15 2019-01-07 15:36:10,387 WARN [cloud.resource.ServerResourceBase] (main:null) (logid:) Incorrect details for private Nic during initialization of ServerResourceBase 2019-01-07 15:36:10,387 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: Unable to configure LibvirtComputingResource This occurs on version 4.11.1 and 4.11.2, and I've seen other threads with this issue on earlier versions that reported downgrading the agent fixed this issue. NIC CONFIGURATION: ifcfg-cloudbr0 DEVICE=cloudbr0 TYPE=Bridge ONBOOT=yes BOOTPROTO=none IPV6INIT=no IPV6_AUTOCONF=no DELAY=5 STP=yes Ifcfg-eno1.8 HWADDR=XX:XX:XX:XX:XX:XX DEVICE=eno1.8 ONBOOT=yes HOTPLUG=no BOOTPROTO=none TYPE=Vlan VLAN=yes BRIDGE=cloudbr0 cloudbr0: flags=4163 mtu 1500 ether txqueuelen 1000 (Ethernet) RX packets 2368 bytes 116316 (113.5 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 3 bytes 270 (270.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eno1.8: flags=4163 mtu 1500 ether txqueuelen 1000 (Ethernet) RX packets 3159 bytes 163465 (159.6 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1752 bytes 91924 (89.7 KiB)
Re: Unable to schedule async job
Hi, Dag, Thank for the reply. Below is my async_job table. Today, I try to delete some VM. Interestingly, I can delete some VM and some are failed withthis error: Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '387' for key 'PRIMARY' But in the async_job table, I don't see id=387 there. I have attached part of the Management Server log wrt the delete VM action. One minor correction. I am actually using cloudstack 4.11.1 instead of 4.9.2. MariaDB [cloud]> select id, job_cmd from async_job order by id; +-+-+ | id | job_cmd | +-+-+ | 275 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 276 | NULL| | 281 | NULL| | 287 | NULL| | 288 | NULL| | 292 | NULL| | 305 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 307 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 309 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 313 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | | 314 | NULL| | 315 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | | 316 | com.cloud.vm.VmWorkStart| | 317 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 319 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 321 | NULL| | 322 | com.cloud.vm.VmWorkStop | | 323 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 327 | com.cloud.vm.VmWorkStart| | 328 | com.cloud.vm.VmWorkStart| | 329 | NULL| | 330 | com.cloud.vm.VmWorkStop | | 332 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 334 | com.cloud.vm.VmWorkStop | | 335 | com.cloud.vm.VmWorkStart| | 336 | com.cloud.vm.VmWorkStart| | 338 | org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin | | 340 | com.cloud.vm.VmWorkStop | | 341 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 342 | com.cloud.vm.VmWorkStop | | 343 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| | 344 | com.cloud.vm.VmWorkStart| | 345 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 346 | com.cloud.vm.VmWorkStop | | 347 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 348 | com.cloud.vm.VmWorkStop | | 349 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| | 350 | com.cloud.vm.VmWorkStart| | 351 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 352 | com.cloud.vm.VmWorkStop | | 354 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 355 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 356 | com.cloud.vm.VmWorkStop | | 357 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots| | 358 | com.cloud.vm.VmWorkStop | | 359 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 360 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 361 | com.cloud.vm.VmWorkStop | | 362 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots| | 363 | com.cloud.vm.VmWorkStop | | 364 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 365 | com.cloud.vm.VmWorkStop | | 366 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots| | 367 | com.cloud.vm.VmWorkStop | | 368 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 369 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 370 |
CloudStack meetup, London March 14
Hi all - happy new year! We've still got a couple of speaking slots to fill, so if you are interested please contact me directly. For more info and to register please see here: https://www.eventbrite.co.uk/e/cloudstack-european-user-group-meetup-tickets-53905627182 Thanks! Best regards, steve.ro...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue
Re: Unable to schedule async job
What does your async_job table say around id>330? Regards, Dag Sonstebo Cloud Architect ShapeBlue On 10/01/2019, 05:34, "Ivan X Yue" wrote: Hi, I am using CloudStack 4.9.2 with KVM hypervisors. Today, I find that the hypervisor is not responding, and therefore I restart it. After that, I find that virtual routers are stopped. When I try to start them, I keep getting "Unable to schedule async job" error. From the management-server.log, I see some exception related to MySQL: Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '333' for key 'PRIMARY' Is there any cleanup that I need to do in the database? How can I do that? Here is the full stacktrace of the exception that I get: 2019-01-09 12:06:31,417 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-3248f525 job-332 ctx-d797e8bd) (logid:d37b7ec3) Unable to schedule async job for command com.cloud.vm.VmWorkStop, unexpected exception. javax.persistence.EntityExistsException: Entity already exists: at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231) at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50) at com.cloud.utils.db.Transaction.execute(Transaction.java:40) at com.cloud.utils.db.Transaction.execute(Transaction.java:47) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231) at com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498) at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600) at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513) at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502) at com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491) at com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253) at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy239.destroyRouter(Unknown Source) at org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd.execute(DestroyRouterCmd.java:103) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
