[
https://issues.apache.org/jira/browse/CLOUDSTACK-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koushik Das resolved CLOUDSTACK-8603.
-------------------------------------
Resolution: Fixed
> Random list VM failures at scale (more than 1000 VMs) when VM has resource
> tags
> -------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-8603
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8603
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.5.0, 4.5.1
> Reporter: Koushik Das
> Assignee: Koushik Das
> Fix For: 4.6.0
>
>
> Random failures seen in list VM API call (VMs having resource tag) with the
> following exception. As can be seen from the stack trace, exception is thrown
> while creating the list response. The test scenario involved doing VM
> creation/deletion/list operations concurrently over a period of time.
> 2015-05-27 00:16:54,239 ERROR [c.c.a.ApiServer]
> (catalina-exec-24:ctx-c122d00c ctx-fcdc0d00 ctx-3df89f23) (logid:6004902e)
> unhandled exception executing api command: [Ljava.lang.String;@76642f16
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> com.cloud.api.query.dao.ResourceTagJoinDaoImpl.searchById(ResourceTagJoinDaoImpl.java:154)
> at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy86.searchById(Unknown Source)
> at com.cloud.api.ApiDBUtils.findResourceTagViewById(ApiDBUtils.java:1636)
> at
> com.cloud.api.query.dao.UserVmJoinDaoImpl.newUserVmResponse(UserVmJoinDaoImpl.java:266)
> at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy176.newUserVmResponse(Unknown Source)
> at com.cloud.api.ApiDBUtils.newUserVmResponse(ApiDBUtils.java:1600)
> at
> com.cloud.api.query.ViewResponseHelper.createUserVmResponse(ViewResponseHelper.java:140)
> at
> com.cloud.api.query.QueryManagerImpl.searchForUserVMs(QueryManagerImpl.java:762)
> at
> org.apache.cloudstack.api.command.user.vm.ListVMsCmd.execute(ListVMsCmd.java:227)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:700)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:525)
> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:282)
> There are 2 part to list VM API:
> a. Get a list of VMs based on search criteria from the user_vm_view view in
> DB.
> b. Create the response records corresponding to each VM obtained as part of
> (a). This record creation involves fetching additional data from other
> views/tables in the DB. One such view is resource_tag_view.
> Now consider the following sequence of events that may lead to the issue:
> a. List VM call fetches a lot of VMs from user_vm_view.
> b. VM response is created for each VM by iterating over the list obtained
> from (a).
> c. Now while (b) is in progress, a VM which is in the list from (a) and not
> yet processed in (b) got destroyed.
> d. Now when response generation happens for destroyed VM, there is a DB query
> to fetch resource tag details from resource_tag_view. This query returns
> empty as the resource tag got deleted from DB as part of VM destroy.
> e. In the resource tag fetching code there is no check for the condition
> mentioned in (d) and so the index out of bound exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
