Re: Problem due to create network
Hi Thomas, I wrote about this global parameter(network.lock.timeout) in previous message. I set it to 2000, but still seeing message "Timed out on acquiring lock account2 . Waited for 300seconds" in logs. пт, 14 июн. 2019 г. в 15:50, Thomas Joseph : > check your settings in the global parameters for: > network.lock.timeout Lock wait timeout (seconds) while implementing network > > > On Fri, Jun 14, 2019 at 1:07 PM Dmitry Berezhnoy > wrote: > > > Somebody else can help? We also increased network.lock.timeout but it was > > useless. > > > > ср, 12 июн. 2019 г. в 00:40, Andrija Panic : > > > > > Based on the " > > > > > > > > > https://github.com/apache/cloudstack/blob/1d05fead49f5c856257a741b07122f5633d2e359/framework/db/src/main/java/com/cloud/utils/db/Merovingian2.java#L121:L154 > > > " > > > I'm not quite sure it's a configurable thing. I don't see any parameter > > > with that value or i.e. half of that... > > > > > > Perhaps someone else can advice > > > > > > > > > On Mon, 10 Jun 2019 at 22:16, Dmitry Berezhnoy > > > wrote: > > > > > > > Yep, we use CloudStack as testing platform for our product and we > want > > to > > > > create VMs and networks ASAP via scripts, so we don't see any > > restriction > > > > for that from physical platform or hypervisor, why CloudStack can't > do > > > > that? We want just find and increase 300 seconds parameters from this > > > > message "Timed out on acquiring lock account2 . Waited for > > 300seconds", > > > > can you help us with it? > > > > > > > > пн, 10 июн. 2019 г. в 21:59, Andrija Panic >: > > > > > > > > > In mgmt log you have 904 createNetwork API calls !!! > > > > > Don't you think it's a bit too much ??? > > > > > > > > > > Check who is nailing your API, since your log looks really funny in > > > that > > > > > regards - either some automation or denial of service thing or > > > something > > > > > else (on client side) really wrong. > > > > > > > > > > Cheers > > > > > > > > > > On Mon, 10 Jun 2019 at 19:25, Dmitry Berezhnoy < > > d.berezh...@emzior.ru> > > > > > wrote: > > > > > > > > > > > Hi Andrija, > > > > > > > > > > > > https://yadi.sk/d/BNOQFeCN86szxw - agent.log > > > > > > https://yadi.sk/d/5UdQ4RExJ-VfmA - management-server.log > > > > > > It is fresh installation with using for some days. > > > > > > > > > > > > Dmitry > > > > > > > > > > > > пн, 10 июн. 2019 г. в 17:44, Andrija Panic < > > andrija.pa...@gmail.com > > > >: > > > > > > > > > > > > > Hi Dmitry, > > > > > > > > > > > > > > if not mistaken, this might have something to do with DB - can > > you > > > > > please > > > > > > > post whole log, on pastebin or similar ? > > > > > > > > > > > > > > Is this a fresh installation (test) or a running production for > > > some > > > > > > time ? > > > > > > > > > > > > > > Andrija > > > > > > > > > > > > > > On Mon, 10 Jun 2019 at 14:55, Dmitry Berezhnoy < > > > > d.berezh...@emzior.ru> > > > > > > > wrote: > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > Just tried cloudstack 4.11.2 and got the error due to create > > > > network. > > > > > > In > > > > > > > > cloudstack logs we got the message: "Timed out on acquiring > > lock > > > > > > account2 > > > > > > > > . Waited for 300seconds" (please read the attachment for > more > > > > > > details). > > > > > > > > Which parameter defines this 300 seconds, where we can > increase > > > it? > > > > > Any > > > > > > > > ideas? > > > > > > > > > > > > > > > > Thanks in advance for the help, > > > > > > > > Dmitry. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > -- > > > > > > Andrija Panić > > > > > >
Re: Problem due to create network
check your settings in the global parameters for: network.lock.timeout Lock wait timeout (seconds) while implementing network On Fri, Jun 14, 2019 at 1:07 PM Dmitry Berezhnoy wrote: > Somebody else can help? We also increased network.lock.timeout but it was > useless. > > ср, 12 июн. 2019 г. в 00:40, Andrija Panic : > > > Based on the " > > > > > https://github.com/apache/cloudstack/blob/1d05fead49f5c856257a741b07122f5633d2e359/framework/db/src/main/java/com/cloud/utils/db/Merovingian2.java#L121:L154 > > " > > I'm not quite sure it's a configurable thing. I don't see any parameter > > with that value or i.e. half of that... > > > > Perhaps someone else can advice > > > > > > On Mon, 10 Jun 2019 at 22:16, Dmitry Berezhnoy > > wrote: > > > > > Yep, we use CloudStack as testing platform for our product and we want > to > > > create VMs and networks ASAP via scripts, so we don't see any > restriction > > > for that from physical platform or hypervisor, why CloudStack can't do > > > that? We want just find and increase 300 seconds parameters from this > > > message "Timed out on acquiring lock account2 . Waited for > 300seconds", > > > can you help us with it? > > > > > > пн, 10 июн. 2019 г. в 21:59, Andrija Panic : > > > > > > > In mgmt log you have 904 createNetwork API calls !!! > > > > Don't you think it's a bit too much ??? > > > > > > > > Check who is nailing your API, since your log looks really funny in > > that > > > > regards - either some automation or denial of service thing or > > something > > > > else (on client side) really wrong. > > > > > > > > Cheers > > > > > > > > On Mon, 10 Jun 2019 at 19:25, Dmitry Berezhnoy < > d.berezh...@emzior.ru> > > > > wrote: > > > > > > > > > Hi Andrija, > > > > > > > > > > https://yadi.sk/d/BNOQFeCN86szxw - agent.log > > > > > https://yadi.sk/d/5UdQ4RExJ-VfmA - management-server.log > > > > > It is fresh installation with using for some days. > > > > > > > > > > Dmitry > > > > > > > > > > пн, 10 июн. 2019 г. в 17:44, Andrija Panic < > andrija.pa...@gmail.com > > >: > > > > > > > > > > > Hi Dmitry, > > > > > > > > > > > > if not mistaken, this might have something to do with DB - can > you > > > > please > > > > > > post whole log, on pastebin or similar ? > > > > > > > > > > > > Is this a fresh installation (test) or a running production for > > some > > > > > time ? > > > > > > > > > > > > Andrija > > > > > > > > > > > > On Mon, 10 Jun 2019 at 14:55, Dmitry Berezhnoy < > > > d.berezh...@emzior.ru> > > > > > > wrote: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > Just tried cloudstack 4.11.2 and got the error due to create > > > network. > > > > > In > > > > > > > cloudstack logs we got the message: "Timed out on acquiring > lock > > > > > account2 > > > > > > > . Waited for 300seconds" (please read the attachment for more > > > > > details). > > > > > > > Which parameter defines this 300 seconds, where we can increase > > it? > > > > Any > > > > > > > ideas? > > > > > > > > > > > > > > Thanks in advance for the help, > > > > > > > Dmitry. > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Andrija Panić > > > > > > > > > > > > > -- > > > > Andrija Panić > > >
Re: Problem due to create network
Somebody else can help? We also increased network.lock.timeout but it was useless. ср, 12 июн. 2019 г. в 00:40, Andrija Panic : > Based on the " > > https://github.com/apache/cloudstack/blob/1d05fead49f5c856257a741b07122f5633d2e359/framework/db/src/main/java/com/cloud/utils/db/Merovingian2.java#L121:L154 > " > I'm not quite sure it's a configurable thing. I don't see any parameter > with that value or i.e. half of that... > > Perhaps someone else can advice > > > On Mon, 10 Jun 2019 at 22:16, Dmitry Berezhnoy > wrote: > > > Yep, we use CloudStack as testing platform for our product and we want to > > create VMs and networks ASAP via scripts, so we don't see any restriction > > for that from physical platform or hypervisor, why CloudStack can't do > > that? We want just find and increase 300 seconds parameters from this > > message "Timed out on acquiring lock account2 . Waited for 300seconds", > > can you help us with it? > > > > пн, 10 июн. 2019 г. в 21:59, Andrija Panic : > > > > > In mgmt log you have 904 createNetwork API calls !!! > > > Don't you think it's a bit too much ??? > > > > > > Check who is nailing your API, since your log looks really funny in > that > > > regards - either some automation or denial of service thing or > something > > > else (on client side) really wrong. > > > > > > Cheers > > > > > > On Mon, 10 Jun 2019 at 19:25, Dmitry Berezhnoy > > > wrote: > > > > > > > Hi Andrija, > > > > > > > > https://yadi.sk/d/BNOQFeCN86szxw - agent.log > > > > https://yadi.sk/d/5UdQ4RExJ-VfmA - management-server.log > > > > It is fresh installation with using for some days. > > > > > > > > Dmitry > > > > > > > > пн, 10 июн. 2019 г. в 17:44, Andrija Panic >: > > > > > > > > > Hi Dmitry, > > > > > > > > > > if not mistaken, this might have something to do with DB - can you > > > please > > > > > post whole log, on pastebin or similar ? > > > > > > > > > > Is this a fresh installation (test) or a running production for > some > > > > time ? > > > > > > > > > > Andrija > > > > > > > > > > On Mon, 10 Jun 2019 at 14:55, Dmitry Berezhnoy < > > d.berezh...@emzior.ru> > > > > > wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > Just tried cloudstack 4.11.2 and got the error due to create > > network. > > > > In > > > > > > cloudstack logs we got the message: "Timed out on acquiring lock > > > > account2 > > > > > > . Waited for 300seconds" (please read the attachment for more > > > > details). > > > > > > Which parameter defines this 300 seconds, where we can increase > it? > > > Any > > > > > > ideas? > > > > > > > > > > > > Thanks in advance for the help, > > > > > > Dmitry. > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > -- > > > > > > Andrija Panić > > > > > > > > -- > > Andrija Panić >
Re: Problem due to create network
Based on the " https://github.com/apache/cloudstack/blob/1d05fead49f5c856257a741b07122f5633d2e359/framework/db/src/main/java/com/cloud/utils/db/Merovingian2.java#L121:L154"; I'm not quite sure it's a configurable thing. I don't see any parameter with that value or i.e. half of that... Perhaps someone else can advice On Mon, 10 Jun 2019 at 22:16, Dmitry Berezhnoy wrote: > Yep, we use CloudStack as testing platform for our product and we want to > create VMs and networks ASAP via scripts, so we don't see any restriction > for that from physical platform or hypervisor, why CloudStack can't do > that? We want just find and increase 300 seconds parameters from this > message "Timed out on acquiring lock account2 . Waited for 300seconds", > can you help us with it? > > пн, 10 июн. 2019 г. в 21:59, Andrija Panic : > > > In mgmt log you have 904 createNetwork API calls !!! > > Don't you think it's a bit too much ??? > > > > Check who is nailing your API, since your log looks really funny in that > > regards - either some automation or denial of service thing or something > > else (on client side) really wrong. > > > > Cheers > > > > On Mon, 10 Jun 2019 at 19:25, Dmitry Berezhnoy > > wrote: > > > > > Hi Andrija, > > > > > > https://yadi.sk/d/BNOQFeCN86szxw - agent.log > > > https://yadi.sk/d/5UdQ4RExJ-VfmA - management-server.log > > > It is fresh installation with using for some days. > > > > > > Dmitry > > > > > > пн, 10 июн. 2019 г. в 17:44, Andrija Panic : > > > > > > > Hi Dmitry, > > > > > > > > if not mistaken, this might have something to do with DB - can you > > please > > > > post whole log, on pastebin or similar ? > > > > > > > > Is this a fresh installation (test) or a running production for some > > > time ? > > > > > > > > Andrija > > > > > > > > On Mon, 10 Jun 2019 at 14:55, Dmitry Berezhnoy < > d.berezh...@emzior.ru> > > > > wrote: > > > > > > > > > Hello, > > > > > > > > > > Just tried cloudstack 4.11.2 and got the error due to create > network. > > > In > > > > > cloudstack logs we got the message: "Timed out on acquiring lock > > > account2 > > > > > . Waited for 300seconds" (please read the attachment for more > > > details). > > > > > Which parameter defines this 300 seconds, where we can increase it? > > Any > > > > > ideas? > > > > > > > > > > Thanks in advance for the help, > > > > > Dmitry. > > > > > > > > > > > > > > > > > -- > > > > > > > > Andrija Panić > > > > > > > > > > > > > -- > > > > Andrija Panić > > > -- Andrija Panić
Re: Problem due to create network
Yep, we use CloudStack as testing platform for our product and we want to create VMs and networks ASAP via scripts, so we don't see any restriction for that from physical platform or hypervisor, why CloudStack can't do that? We want just find and increase 300 seconds parameters from this message "Timed out on acquiring lock account2 . Waited for 300seconds", can you help us with it? пн, 10 июн. 2019 г. в 21:59, Andrija Panic : > In mgmt log you have 904 createNetwork API calls !!! > Don't you think it's a bit too much ??? > > Check who is nailing your API, since your log looks really funny in that > regards - either some automation or denial of service thing or something > else (on client side) really wrong. > > Cheers > > On Mon, 10 Jun 2019 at 19:25, Dmitry Berezhnoy > wrote: > > > Hi Andrija, > > > > https://yadi.sk/d/BNOQFeCN86szxw - agent.log > > https://yadi.sk/d/5UdQ4RExJ-VfmA - management-server.log > > It is fresh installation with using for some days. > > > > Dmitry > > > > пн, 10 июн. 2019 г. в 17:44, Andrija Panic : > > > > > Hi Dmitry, > > > > > > if not mistaken, this might have something to do with DB - can you > please > > > post whole log, on pastebin or similar ? > > > > > > Is this a fresh installation (test) or a running production for some > > time ? > > > > > > Andrija > > > > > > On Mon, 10 Jun 2019 at 14:55, Dmitry Berezhnoy > > > wrote: > > > > > > > Hello, > > > > > > > > Just tried cloudstack 4.11.2 and got the error due to create network. > > In > > > > cloudstack logs we got the message: "Timed out on acquiring lock > > account2 > > > > . Waited for 300seconds" (please read the attachment for more > > details). > > > > Which parameter defines this 300 seconds, where we can increase it? > Any > > > > ideas? > > > > > > > > Thanks in advance for the help, > > > > Dmitry. > > > > > > > > > > > > > -- > > > > > > Andrija Panić > > > > > > > > -- > > Andrija Panić >
Re: Problem due to create network
In mgmt log you have 904 createNetwork API calls !!! Don't you think it's a bit too much ??? Check who is nailing your API, since your log looks really funny in that regards - either some automation or denial of service thing or something else (on client side) really wrong. Cheers On Mon, 10 Jun 2019 at 19:25, Dmitry Berezhnoy wrote: > Hi Andrija, > > https://yadi.sk/d/BNOQFeCN86szxw - agent.log > https://yadi.sk/d/5UdQ4RExJ-VfmA - management-server.log > It is fresh installation with using for some days. > > Dmitry > > пн, 10 июн. 2019 г. в 17:44, Andrija Panic : > > > Hi Dmitry, > > > > if not mistaken, this might have something to do with DB - can you please > > post whole log, on pastebin or similar ? > > > > Is this a fresh installation (test) or a running production for some > time ? > > > > Andrija > > > > On Mon, 10 Jun 2019 at 14:55, Dmitry Berezhnoy > > wrote: > > > > > Hello, > > > > > > Just tried cloudstack 4.11.2 and got the error due to create network. > In > > > cloudstack logs we got the message: "Timed out on acquiring lock > account2 > > > . Waited for 300seconds" (please read the attachment for more > details). > > > Which parameter defines this 300 seconds, where we can increase it? Any > > > ideas? > > > > > > Thanks in advance for the help, > > > Dmitry. > > > > > > > > > -- > > > > Andrija Panić > > > -- Andrija Panić
Re: Problem due to create network
Hi Andrija, https://yadi.sk/d/BNOQFeCN86szxw - agent.log https://yadi.sk/d/5UdQ4RExJ-VfmA - management-server.log It is fresh installation with using for some days. Dmitry пн, 10 июн. 2019 г. в 17:44, Andrija Panic : > Hi Dmitry, > > if not mistaken, this might have something to do with DB - can you please > post whole log, on pastebin or similar ? > > Is this a fresh installation (test) or a running production for some time ? > > Andrija > > On Mon, 10 Jun 2019 at 14:55, Dmitry Berezhnoy > wrote: > > > Hello, > > > > Just tried cloudstack 4.11.2 and got the error due to create network. In > > cloudstack logs we got the message: "Timed out on acquiring lock account2 > > . Waited for 300seconds" (please read the attachment for more details). > > Which parameter defines this 300 seconds, where we can increase it? Any > > ideas? > > > > Thanks in advance for the help, > > Dmitry. > > > > > -- > > Andrija Panić >
Re: Problem due to create network
Hi Dmitry, if not mistaken, this might have something to do with DB - can you please post whole log, on pastebin or similar ? Is this a fresh installation (test) or a running production for some time ? Andrija On Mon, 10 Jun 2019 at 14:55, Dmitry Berezhnoy wrote: > Hello, > > Just tried cloudstack 4.11.2 and got the error due to create network. In > cloudstack logs we got the message: "Timed out on acquiring lock account2 > . Waited for 300seconds" (please read the attachment for more details). > Which parameter defines this 300 seconds, where we can increase it? Any > ideas? > > Thanks in advance for the help, > Dmitry. > -- Andrija Panić
Problem due to create network
Hello, Just tried cloudstack 4.11.2 and got the error due to create network. In cloudstack logs we got the message: "Timed out on acquiring lock account2 . Waited for 300seconds" (please read the attachment for more details). Which parameter defines this 300 seconds, where we can increase it? Any ideas? Thanks in advance for the help, Dmitry. 2019-06-10 14:24:26,432 DEBUG [c.c.a.ApiServlet] (qtp895947612-1041:ctx-5f19e1ca) (logid:d1cee349) ===START=== 10.10.12.10 -- GET apikey=xRZ8vfHekftVn43EKETrcVlgJu-Vcd7c47-GjsiEIlGhF-D428h5HruTGaN21uDuNsYTsB-NTW8QCTJM8sz2yQ&name= redistr-200-49308-mac-change-0-0&networkofferingid=3b96bbc8-1102-48de-867b-a4162988db84&gateway=10.200.1.1&netmask=255.255.255.0&command=createNetwork&displaytext=redistr-200-49308-mac-change-0-0&zoneid=d103e528-c0b0-471e-ad55-2b99 c47c4dee&response=json&signature=QTewi3D%2Bo%2Ft4jfoh9UOTLcLBPhk%3D 2019-06-10 14:24:26,439 DEBUG [c.c.a.ApiServer] (qtp895947612-1041:ctx-5f19e1ca ctx-2416e4bc) (logid:d1cee349) CIDRs from which account 'Acct[d9d375fe-87a1-11e9-a390-0cc47aaa7eb0-admin]' is allowed to perform API calls: 0.0.0.0/0,: :/0 2019-06-10 14:29:28,229 WARN [c.c.u.d.Merovingian2] (qtp895947612-1041:ctx-5f19e1ca ctx-2416e4bc ctx-da492f0d) (logid:d1cee349) Timed out on acquiring lock account2 . Waited for 300seconds com.cloud.utils.exception.CloudRuntimeException: Timed out on acquiring lock account2 . Waited for 300seconds at com.cloud.utils.db.Merovingian2.acquire(Merovingian2.java:151) at com.cloud.utils.db.TransactionLegacy.lock(TransactionLegacy.java:386) at com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1048) at com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1039) at sun.reflect.GeneratedMethodAccessor152.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy35.acquireInLockTable(Unknown Source) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:636) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$10.doInTransaction(NetworkOrchestrator.java:2357) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$10.doInTransaction(NetworkOrchestrator.java:2308) at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50) at com.cloud.utils.db.Transaction.execute(Transaction.java:40) at com.cloud.utils.db.Transaction.execute(Transaction.java:47) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createGuestNetwork(NetworkOrchestrator.java:2308) at com.cloud.network.NetworkServiceImpl$4.doInTransaction(NetworkServiceImpl.java:1383) at com.cloud.network.NetworkServiceImpl$4.doInTransaction(NetworkServiceImpl.java:1331) at com.cloud.utils.db.Transaction.execute(Transaction.java:40) at com.cloud.network.NetworkServiceImpl.commitNetwork(NetworkServiceImpl.java:1331) at com.cloud.network.NetworkServiceImpl.createGuestNetwork(NetworkServiceImpl.java:1294) at sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventIn
