On Jul 14, 2011, at 12:16 AM, David Lutterkort (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/DTACLOUD-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064913#comment-13064913 > ] > > David Lutterkort commented on DTACLOUD-58: > ------------------------------------------ > > The 'heartbeat' isn't coming from deltacloud, but from aeolus; reloading the > list of all VM's every 30s is way too often. > > At the same time, I don't understand why it takes RHEV-M this long to list 8 > VM's > > I am afraid deltacloud is just the messenger in this issue
Right, we're just forwarding requests to RHEV-M Rest API written on top of PowerShell interface to RHEV-M. You can see how much it takes for backend to process your request by looking at response headers. Search for 'X-Backend-Runtime:' header. -- MIchal > >> api/instance query performance with 8 vms in rhevm >> -------------------------------------------------- >> >> Key: DTACLOUD-58 >> URL: https://issues.apache.org/jira/browse/DTACLOUD-58 >> Project: DeltaCloud >> Issue Type: Bug >> Environment: Aeolus-conductor configured for rhevm provider >> Rhevm running in Windows 2008 R2 Server with 1.5GB memory VM (running on a >> separate blade server) >> [root@hp-dl2x170g6-01 ~]# rpm -qa | egrep 'aeolus|deltacloud' | sort >> aeolus-all-0.3.0-0.el6.20110711131044git5bc7abf.noarch >> aeolus-conductor-0.3.0-0.el6.20110711131044git5bc7abf.noarch >> aeolus-conductor-daemons-0.3.0-0.el6.20110711131044git5bc7abf.noarch >> aeolus-conductor-doc-0.3.0-0.el6.20110711131044git5bc7abf.noarch >> aeolus-configure-2.0.1-0.el6.20110708134115gitab1e6dc.noarch >> condor-deltacloud-gahp-7.6.0-5dcloud.el6.x86_64 >> deltacloud-core-0.3.9999-1308927004.el6.noarch >> libdeltacloud-0.9-1.el6.x86_64 >> rubygem-aeolus-cli-0.0.1-1.el6.20110711131044git5bc7abf.noarch >> rubygem-deltacloud-client-0.3.1-1.el6.noarch >> [root@hp-dl2x170g6-01 ~]# >> Reporter: dave johnson >> Assignee: David Lutterkort >> Attachments: rhevm.perf.log >> >> >> I noticed that when I had over 7 rhev-h vm's running within rhevm, the CPU >> was running > 90% and the api console was practically scrolling >> continuously. I looked at the deltacloud-core log file for rhevm and >> noticed that the query time of /api/instances were taking > 45 seconds to >> complete. I started over adding a single vm every five minutes and noticed >> that there is a "heartbeat" from deltacloud-core every 30 seconds, if >> querying the API is taking longer than the heartbeat interval, >> deltacloud-core's heartbeart is causing performance issues with the rhevm >> api. >> With 8 vm's running, deltacloud-core api/instabces query were timing out and >> throwing timeout exceptions. >> # instances deployed through conductor >> # instance starts in deltacloud-core log >> 127.0.0.1 - - [13/Jul/2011 16:13:15] "POST >> /api/instances/7b33e17e-3563-4717-b8e4-e5d5f454a240/start HTTP/1.1" 204 - >> 2.3925 >> 127.0.0.1 - - [13/Jul/2011 16:20:50] "POST >> /api/instances/b24424ae-ab1e-4974-ba97-c0e4a8a2a779/start HTTP/1.1" 204 - >> 2.0416 >> 127.0.0.1 - - [13/Jul/2011 16:26:01] "POST >> /api/instances/20a5dcad-b053-4267-9e5b-46f40f9d42c7/start HTTP/1.1" 204 - >> 3.9677 >> 127.0.0.1 - - [13/Jul/2011 16:31:48] "POST >> /api/instances/e79f7b0e-7cc8-4426-88d7-3a3e1728fbb6/start HTTP/1.1" 204 - >> 13.0901 >> 127.0.0.1 - - [13/Jul/2011 16:35:55] "POST >> /api/instances/5327ddc2-8a95-43c9-8b00-80e28fa1a16b/start HTTP/1.1" 204 - >> 4.2185 >> 127.0.0.1 - - [13/Jul/2011 16:41:23] "POST >> /api/instances/2d8624a7-ecd5-4e7a-a479-f2ae45f96a70/start HTTP/1.1" 204 - >> 7.6413 >> 127.0.0.1 - - [13/Jul/2011 16:46:33] "POST >> /api/instances/0b67d735-cd54-4a12-ba43-d708b018e6d0/start HTTP/1.1" 204 - >> 6.6535 >> 127.0.0.1 - - [13/Jul/2011 16:52:26] "POST >> /api/instances/d16a2597-0916-419f-83cd-86e41dc08ca6/start HTTP/1.1" 204 - >> 20.1581 >> # api/instance load times climbing >> 127.0.0.1 - - [13/Jul/2011 16:15:22] "GET /api/instances HTTP/1.1" 200 1223 >> 3.2926 >> 127.0.0.1 - - [13/Jul/2011 16:20:12] "GET /api/instances HTTP/1.1" 200 1223 >> 3.4131 >> 127.0.0.1 - - [13/Jul/2011 16:25:17] "GET /api/instances HTTP/1.1" 200 2381 >> 6.8360 >> 127.0.0.1 - - [13/Jul/2011 16:30:54] "GET /api/instances HTTP/1.1" 200 3539 >> 9.8815 >> 127.0.0.1 - - [13/Jul/2011 16:35:17] "GET /api/instances HTTP/1.1" 200 4697 >> 8.3200 >> 127.0.0.1 - - [13/Jul/2011 16:40:51] "GET /api/instances HTTP/1.1" 200 5855 >> 21.5512 >> 127.0.0.1 - - [13/Jul/2011 16:45:00] "GET /api/instances HTTP/1.1" 200 7013 >> 24.4277 >> 127.0.0.1 - - [13/Jul/2011 16:50:05] "GET /api/instances HTTP/1.1" 200 8171 >> 35.1951 >> 127.0.0.1 - - [13/Jul/2011 16:50:07] "GET /api/instances HTTP/1.1" 200 8171 >> 34.3853 >> 127.0.0.1 - - [13/Jul/2011 16:50:14] "GET /api/instances HTTP/1.1" 200 8171 >> 34.7330 >> 127.0.0.1 - - [13/Jul/2011 16:50:19] "GET /api/instances HTTP/1.1" 200 8171 >> 34.0917 >> 127.0.0.1 - - [13/Jul/2011 16:50:32] "GET /api/instances HTTP/1.1" 200 8171 >> 34.0356 >> 127.0.0.1 - - [13/Jul/2011 16:50:57] "GET /api/instances HTTP/1.1" 200 8171 >> 47.4070 >> 127.0.0.1 - - [13/Jul/2011 16:51:05] "GET /api/instances HTTP/1.1" 200 8171 >> 50.6648 >> 127.0.0.1 - - [13/Jul/2011 16:51:08] "GET /api/instances HTTP/1.1" 200 8171 >> 52.6800 >> 127.0.0.1 - - [13/Jul/2011 16:51:09] "GET /api/instances HTTP/1.1" 200 8171 >> 53.5951 >> 127.0.0.1 - - [13/Jul/2011 16:51:29] "GET /api/instances HTTP/1.1" 200 8171 >> 53.6821 >> 127.0.0.1 - - [13/Jul/2011 16:51:34] "GET /api/instances HTTP/1.1" 200 8171 >> 57.5881 >> 127.0.0.1 - - [13/Jul/2011 16:51:35] "GET /api/instances HTTP/1.1" 200 8171 >> 51.4359 >> 127.0.0.1 - - [13/Jul/2011 16:51:38] "GET /api/instances HTTP/1.1" 200 8171 >> 48.9456 >> 127.0.0.1 - - [13/Jul/2011 16:51:42] "GET /api/instances HTTP/1.1" 200 8171 >> 39.6616 >> 127.0.0.1 - - [13/Jul/2011 16:52:01] "GET /api/instances HTTP/1.1" 200 9454 >> 33.8931 >> 127.0.0.1 - - [13/Jul/2011 16:52:13] "GET /api/instances HTTP/1.1" 200 9454 >> 37.3036 >> 127.0.0.1 - - [13/Jul/2011 16:52:20] "GET /api/instances HTTP/1.1" 200 9454 >> 41.3128 >> 127.0.0.1 - - [13/Jul/2011 16:52:58] "GET /api/instances HTTP/1.1" 200 9454 >> 59.2965 >> 127.0.0.1 - - [13/Jul/2011 16:53:04] "GET /api/instances HTTP/1.1" 500 190 >> 60.0303 >> 127.0.0.1 - - [13/Jul/2011 16:53:05] "GET /api/instances HTTP/1.1" 500 190 >> 60.0688 > > -- > This message is automatically generated by JIRA. > For more information on JIRA, see: http://www.atlassian.com/software/jira > > ---------------------------------------------------------------------- Michal Fojtik, Software Engineer, Red Hat Czech [email protected] Deltacloud API: http://deltacloud.org
