[
https://issues.apache.org/jira/browse/CLOUDSTACK-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736186#comment-15736186
]
ASF subversion and git services commented on CLOUDSTACK-9626:
-------------------------------------------------------------
Commit c89198f4c65e74a27a5586e4e18b0e71b164a904 in cloudstack's branch
refs/heads/4.9 from [[email protected]]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=c89198f ]
Merge pull request #1796 from sudhansu7/CLOUDSTACK-9626
CLOUDSTACK-9626: Instance fails to start after unsuccesful computeISSUE
============
Instance fails to start after unsuccesful compute offering upgrade.
TROUBLESHOOTING
==================
We observed VM instance get compute values "cpuNumber","cpuSpeed","memory"
removed from table "user_vm_details", which cause instance fail to startup next
time on XenServer
`mysql> select * from user_vm_details where vm_id=10;
--------------------------------------------------------------------------------------------------
id vm_id name value display
--------------------------------------------------------------------------------------------------
218 10 platform viridian:true;acpi:1;apic:true;pae:true;nx:true
1
219 10 hypervisortoolsversion xenserver56 1
220 10 Message.ReservedCapacityFreed.Flag true 1
--------------------------------------------------------------------------------------------------
3 rows in set (0.00 sec)`
`2016-11-29 06:49:03,667 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-12:ctx-49c25b1d job-125) (logid:114a2f1b) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin
java.lang.NullPointerException
at
com.cloud.vm.UserVmManagerImpl.upgradeRunningVirtualMachine(UserVmManagerImpl.java:1664)
at
com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1631)
at
com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1561)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy197.upgradeVirtualMachine(Unknown Source)
at
org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin.execute(ScaleVMCmdByAdmin.java:48)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)`
REPRO STEPS
==================
1. Set global setting enable.dynamic.scale.vm to true
2. Create a custom Compute Offerings A
3. Create a VM instance apply A, ie. cpuNumber=1,cpuSpeed=1000,memory=512M
4. Create another custom Compute Offerings B
5. Change service offering to B, ie. cpuNumber=2,cpuSpeed=2000,memory=4096M
(ensure 4 times over previous memory size), then you will encounter scaling
failed
6. Stop VM instance , you will never startup again
EXPECTED BEHAVIOR
==================
Succeed Startup VM instance
ACTUAL BEHAVIOR
==================
Fail to start instance
RCA:
The ROLLBACK does not take care of restoring old service offering details. In
case failure we are removing the new service offering details but restoring old
service offering details is missing.
Before Fix:
`user_vm_details before upgrade.
mysql> select * from user_vm_details where vm_id =9;
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| id | vm_id | name | value
| display |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| 118 | 9 | platform |
viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 |
| 119 | 9 | hypervisortoolsversion | xenserver56
| 1 |
| 120 | 9 | Message.ReservedCapacityFreed.Flag | false
| 1 |
| 121 | 9 | cpuNumber | 1
| 1 |
| 122 | 9 | cpuSpeed | 1000
| 1 |
| 123 | 9 | memory | 256
| 1 |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
6 rows in set (0.00 sec)
user_vm_details after unsuccessful upgrade.
mysql> select * from user_vm_details where vm_id =9;
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| id | vm_id | name | value
| display |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| 133 | 9 | platform |
viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 |
| 134 | 9 | hypervisortoolsversion | xenserver56
| 1 |
| 135 | 9 | Message.ReservedCapacityFreed.Flag | false
| 1 |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
3 rows in set (0.00 sec)`
After fix:
`
mysql> select * from user_vm_details where vm_id =9;
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| id | vm_id | name | value
| display |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| 166 | 9 | cpuNumber | 1
| 1 |
| 167 | 9 | platform |
viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 |
| 168 | 9 | cpuSpeed | 1000
| 1 |
| 169 | 9 | Message.ReservedCapacityFreed.Flag | false
| 1 |
| 170 | 9 | memory | 256
| 1 |
| 171 | 9 | hypervisortoolsversion | xenserver56
| 1 |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
6 rows in set (0.00 sec)
`
* pr/1796:
CLOUDSTACK-9626: Instance fails to start after unsuccesful compute offering
upgrade.
Signed-off-by: Rohit Yadav <[email protected]>
> Instance fails to start after unsuccesful compute offering upgrade.
> -------------------------------------------------------------------
>
> Key: CLOUDSTACK-9626
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9626
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.8.0
> Reporter: Sudhansu Sahu
> Assignee: Sudhansu Sahu
>
> ISSUE
> ============
> Instance fails to start after unsuccesful compute offering upgrade.
>
> TROUBLESHOOTING
> ==================
> We observed VM instance get compute values "cpuNumber","cpuSpeed","memory"
> removed from table "user_vm_details", which cause instance fail to startup
> next time on XenServer
> mysql> select * from user_vm_details where vm_id=10;
> +-----+-------+------------------------------------+-------------------------------------------------+---------+
> | id | vm_id | name | value
> | display |
> +-----+-------+------------------------------------+-------------------------------------------------+---------+
> | 218 | 10 | platform |
> viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 |
> | 219 | 10 | hypervisortoolsversion | xenserver56
> | 1 |
> | 220 | 10 | Message.ReservedCapacityFreed.Flag | true
> | 1 |
> +-----+-------+------------------------------------+-------------------------------------------------+---------+
> 3 rows in set (0.00 sec)
>
> Unexpected exception while executing
> org.apache.cloudstack.api.command.user.vm.ScaleVMCmd
> java.lang.NullPointerException
> at
> com.cloud.vm.UserVmManagerImpl.upgradeStoppedVirtualMachine(UserVmManagerImpl.java:953)
> at
> com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1331)
> at
> com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1271)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy169.upgradeVirtualMachine(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.vm.ScaleVMCmd.execute(ScaleVMCmd.java:127)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:167)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:97)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:543)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:50)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:47)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:500)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Select by
> Id SQL: SELECT user_vm_details.id, user_vm_details.vm_id,
> user_vm_details.name, user_vm_details.value, user_vm_details.display FROM
> user_vm_details WHERE user_vm_details.id = ?
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Table
> References: user_vm_details
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Insert
> SQLs:
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) INSERT
> INTO user_vm_details (user_vm_details.vm_id, user_vm_details.name,
> user_vm_details.value, user_vm_details.display) VALUES (?, ?, ?, ?)
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Delete SQLs
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) DELETE
> FROM user_vm_details WHERE user_vm_details.id= ?
> REPRO STEPS
> ==================
> 1. Set global setting enable.dynamic.scale.vm to true
> 2. Create a custom Compute Offerings A
> 3. Create a VM instance apply A, ie. cpuNumber=1,cpuSpeed=1000,memory=512M
> 4. Create another custom Compute Offerings B
> 5. Change service offering to B, ie. cpuNumber=2,cpuSpeed=2000,memory=4096M
> (ensure 4 times over previous memory size), then you will encounter scaling
> failed
> 6. Stop VM instance , you will never startup again
> EXPECTED BEHAVIOR
> ==================
> Succeed Startup VM instance
>
> ACTUAL BEHAVIOR
> ==================
> Fail to start instance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
