Koushik Das created CLOUDSTACK-8603:
---------------------------------------
Summary: Random list VM failures at scale (more than 1000 VMs)
when VM has resource tags
Key: CLOUDSTACK-8603
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8603
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.5.1, 4.5.0
Reporter: Koushik Das
Assignee: Koushik Das
Fix For: 4.6.0
Random failures seen in list VM API call (VMs having resource tag) with the
following exception. As can be seen from the stack trace, exception is thrown
while creating the list response. The test scenario involved doing VM
creation/deletion/list operations concurrently over a period of time.
2015-05-27 00:16:54,239 ERROR [c.c.a.ApiServer] (catalina-exec-24:ctx-c122d00c
ctx-fcdc0d00 ctx-3df89f23) (logid:6004902e) unhandled exception executing api
command: [Ljava.lang.String;@76642f16
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
com.cloud.api.query.dao.ResourceTagJoinDaoImpl.searchById(ResourceTagJoinDaoImpl.java:154)
at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy86.searchById(Unknown Source)
at com.cloud.api.ApiDBUtils.findResourceTagViewById(ApiDBUtils.java:1636)
at
com.cloud.api.query.dao.UserVmJoinDaoImpl.newUserVmResponse(UserVmJoinDaoImpl.java:266)
at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy176.newUserVmResponse(Unknown Source)
at com.cloud.api.ApiDBUtils.newUserVmResponse(ApiDBUtils.java:1600)
at
com.cloud.api.query.ViewResponseHelper.createUserVmResponse(ViewResponseHelper.java:140)
at
com.cloud.api.query.QueryManagerImpl.searchForUserVMs(QueryManagerImpl.java:762)
at
org.apache.cloudstack.api.command.user.vm.ListVMsCmd.execute(ListVMsCmd.java:227)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:700)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:525)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:282)
There are 2 part to list VM API:
a. Get a list of VMs based on search criteria from the user_vm_view view in DB.
b. Create the response records corresponding to each VM obtained as part of
(a). This record creation involves fetching additional data from other
views/tables in the DB. One such view is resource_tag_view.
Now consider the following sequence of events that may lead to the issue:
a. List VM call fetches a lot of VMs from user_vm_view.
b. VM response is created for each VM by iterating over the list obtained from
(a).
c. Now while (b) is in progress, a VM which is in the list from (a) and not yet
processed in (b) got destroyed.
d. Now when response generation happens for destroyed VM, there is a DB query
to fetch resource tag details from resource_tag_view. This query returns empty
as the resource tag got deleted from DB as part of VM destroy.
e. In the resource tag fetching code there is no check for the condition
mentioned in (d) and so the index out of bound exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
