[
https://issues.apache.org/jira/browse/CLOUDSTACK-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457655#comment-13457655
]
Rohit Yadav commented on CLOUDSTACK-100:
----------------------------------------
@Sadhu: The bugs (you mentioned) suggest that the MySQL server is unable to
serve under such load (capacity checking queries are intensive)
http://bugs.cloudstack.org/browse/CS-16309
http://bugs.cloudstack.org/browse/CS-10501
Due to inadequate memory, my hypothesis is that the sql server process has many
page error, swap ins/outs, hence slow response from the mysql-connector. Same
goes for mgmt server. If the minimum requirement are not met for a process, it
is bound to fail. (for example, trying to run Battlefield3 on 256MB RAM and
saying that it does not work hence it's a bug is ridiculous :)
So, this bug ticket is invalid. I'm closing the ticket.
But just to be sure, let's create a new test environment and reopen or file a
new bug if this fails;
1. Find out the minimum memory requirement for MySQL server and create a VM
with that memory and host mgmt server and mysql on different VMs.
2. Perform the operations as in this bug.
3. If the bug is caught, perform it at least couple of times. If the bug is
found consistently, then we've a problem.
> capacityChecker throws exception in specific scenario
> -----------------------------------------------------
>
> Key: CLOUDSTACK-100
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-100
> Project: CloudStack
> Issue Type: Bug
> Components: Management Server
> Affects Versions: pre-4.0.0
> Environment: Git Revision: 03df2fa9dd45c938f72cd1866044b09d1b0cc978
> Git URL: https://git-wip-us.apache.org/repos/asf/incubator-cloudstack.gi
> Reporter: sadhu suresh
> Assignee: Rohit Yadav
> Fix For: pre-4.0.0
>
> Attachments: management-server.log.2012-09-12.gz
>
>
> Runtime exception noticed during zone creation on a setup where Management
> server installed on a VM with less memory resources
> Steps:
> 1.Install new build on rhel6.3 OS
> 2.make sure your Ms has less memory capacity(i.e In the current environment
> Management server heartbeat takes too long to finish)
> 3.try to create advance zone with xen as hypervior and observe the behavior
> Actual result:
> Noticed exception in the log while its adding primary storage during zone
> creation operation and it took long time to finish addZone operation due to
> communication took long time to finish between host and Management server.
> Capacity Checker throws runtime exception caused by some sql exception...and
> exception says "you have error in SQL syntax"
> *********************
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You
> have an error in your SQL syntax; check the manual that corresponds to your
> MySQL server version for the right syntax to use near 'B' at line 1
> **********************
> Content of management log:
> 2012-09-12 16:34:44,074 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Management server heartbeat takes too long to
> finish. profiler: Done. Duration: 8837ms, profilerHeartbeatUpdate: Done.
> Duration: 3455ms, profilerPeerScan: Done. Duration: 5382ms, profilerAgentLB:
> Done. Duration: 0ms
> 2012-09-12 16:34:49,824 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Peer scan takes too long to finish. profiler:
> Done. Duration: 4631ms, profilerQueryActiveList: Done. Duration: 2637ms,
> profilerSyncClusterInfo: Done. Duration: 1994ms, profilerInvalidatedNodeList:
> Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms
> 2012-09-12 16:34:52,543 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Management server heartbeat takes too long to
> finish. profiler: Done. Duration: 4699ms, profilerHeartbeatUpdate: Done.
> Duration: 68ms, profilerPeerScan: Done. Duration: 4631ms, profilerAgentLB:
> Done. Duration: 0ms
> 2012-09-12 16:34:54,854 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 routers.
> 2012-09-12 16:34:56,316 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Management server heartbeat takes too long to
> finish. profiler: Done. Duration: 2112ms, profilerHeartbeatUpdate: Done.
> Duration: 704ms, profilerPeerScan: Done. Duration: 1408ms, profilerAgentLB:
> Done. Duration: 0ms
> 2012-09-12 16:35:01,183 INFO [agent.manager.AgentMonitor]
> (AgentMonitor:null) Found the following agents behind on ping: [1]
> 2012-09-12 16:35:01,276 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Peer scan takes too long to finish. profiler:
> Done. Duration: 4690ms, profilerQueryActiveList: Done. Duration: 4394ms,
> profilerSyncClusterInfo: Done. Duration: 296ms, profilerInvalidatedNodeList:
> Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms
> 2012-09-12 16:35:01,634 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Management server heartbeat takes too long to
> finish. profiler: Done. Duration: 5318ms, profilerHeartbeatUpdate: Done.
> Duration: 265ms, profilerPeerScan: Done. Duration: 5053ms, profilerAgentLB:
> Done. Duration: 0ms
> 2012-09-12 16:35:01,874 ERROR [cloud.alert.AlertManagerImpl]
> (CapacityChecker:null) Exception in CapacityChecker
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> org.apache.commons.dbcp.DelegatingPreparedStatement@a9ee006
> at
> com.cloud.utils.db.GenericDaoBase.customSearchIncludingRemoved(GenericDaoBase.java:479)
> at
> com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:34)
> at
> com.cloud.capacity.dao.CapacityDaoImpl.findCapacityBy(CapacityDaoImpl.java:355)
> at
> com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:34)
> at
> com.cloud.alert.AlertManagerImpl.checkForAlerts(AlertManagerImpl.java:473)
> at
> com.cloud.alert.AlertManagerImpl$CapacityChecker.run(AlertManagerImpl.java:442)
> at java.util.TimerThread.mainLoop(Timer.java:534)
> at java.util.TimerThread.run(Timer.java:484)
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You
> have an error in your SQL syntax; check the manual that corresponds to your
> MySQL server version for the right syntax to use near 'B' at line 1
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
> at
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2318)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:97)
> at
> com.cloud.utils.db.GenericDaoBase.customSearchIncludingRemoved(GenericDaoBase.java:459)
> ... 15 more
> 2012-09-12 16:35:33,894 DEBUG [cloud.alert.AlertManagerImpl]
> (CapacityChecker:null) Running Capacity Checker ...
> 2012-09-12 16:35:31,364 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 routers.
> 2012-09-12 16:35:31,191 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-3:null) StorageCollector is running...
> 2012-09-12 16:35:31,191 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-2:null) VmStatsCollector is running...
> 2012-09-12 16:35:15,394 DEBUG [agent.manager.AgentAttache]
> (AgentTaskPool-1:null) Seq 1-1550843917: Waiting some more time because this
> is the current command
> 2012-09-12 16:35:13,612 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-9:null) Ping from 1
> 2012-09-12 16:35:08,496 DEBUG [cloud.host.Status] (AgentMonitor:null) Ping
> timeout for host 1, do invstigation
> 2012-09-12 16:35:05,456 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Peer scan takes too long to finish. profiler:
> Done. Duration: 3620ms, profilerQueryActiveList: Done. Duration: 188ms,
> profilerSyncClusterInfo: Done. Duration: 3432ms, profilerInvalidatedNodeList:
> Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms
> 2012-09-12 16:35:01,875 ERROR [cloud.api.ApiDispatcher]
> (catalina-exec-13:null) Exception while executing ListCapacityCmd:
> java.lang.NullPointerException
> at
> com.cloud.capacity.dao.CapacityDaoImpl.findCapacityBy(CapacityDaoImpl.java:319)
> at
> com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:34)
> at
> com.cloud.server.ManagementServerImpl.listCapacities(ManagementServerImpl.java:2216)
> at
> com.cloud.api.commands.ListCapacityCmd.execute(ListCapacityCmd.java:129)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:543)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:422)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:63)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Expected result:
> AddZone should be completed with out any errors and exception should be
> caught and alert the user with proper information.
> should not get any syntax error exception like MySQLSyntaxErrorExceptions
> seen above logs.
> Note:Not seen the problem when increase the memory on VM from 512 to
> 2GB.also did xe-toolstack-restart ... on xen host.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
