Sanjeev N created CLOUDSTACK-5649:
-------------------------------------
Summary: [Hyper-v] Putting host in maintenance mode does not start
the guest vms and VRs running on it on another available host in the cluster
Key: CLOUDSTACK-5649
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5649
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Hypervisor Controller, Management Server
Affects Versions: 4.3.0
Environment: Latest build from 4.3 branch with
commit:ddd27b3f483319787f7509d13d30b976620831e8
Hypervisor: Hyperv
Storage: SMB for both primary and secondary
Reporter: Sanjeev N
Priority: Critical
Fix For: 4.3.0
[Hyper-v] Putting host in maintenance mode does not migrate the guest vms and
VRs running on it to another available host in the cluster
Steps to Reproduce:
================
1.Bring up CS in advanced zone with two or more hosts in the cluster with
shared storage using SMB for both primary and secondary
2.Deploy few vms(Non-ha enabled) such that vms are distributed across both the
hosts
3.Put one host in maintenance mode
Expected Result:
==============
Host should go to maintenance mode and all the vms running on it should be
migrated to another host
Actual Result:
===========
Host transitioned into maintenance mode but all the vms running on it were not
migrated to another host
In case of host failure only non-HAed vms should be stopped but when the host
is put in maintenance mode from CS all the vms running on it should be migrated
and should be started on another available hosts in the same cluster.
2013-12-26 16:09:18,498 DEBUG [c.c.a.ApiServlet]
(catalina-exec-20:ctx-06e2a7d7) ===START=== 10.146.0.134 -- GET
command=prepareHostForMaintenance&id=bd4e26d2-effb-4c2a-a22a-137a5d91225a&response=json&sessionkey=d%2Bsyfd%2FWn2mU%2Bt9TnFJ4cyRpkbY%3D&_=1388054354967
2013-12-26 16:09:18,645 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-20:ctx-06e2a7d7 ctx-bf5e3753) submit async job-24, details:
AsyncJobVO {id:24, userId: 2, accountId: 2, instanceType: Host, instanceId: 1,
cmd: org.apache.cloudstack.api.command.admin.host.PrepareForMaintenanceCmd,
cmdInfo:
{"response":"json","id":"bd4e26d2-effb-4c2a-a22a-137a5d91225a","sessionkey":"d+syfd/Wn2mU+t9TnFJ4cyRpkbY\u003d","cmdEventType":"MAINT.PREPARE","ctxUserId":"2","httpmethod":"GET","_":"1388054354967","ctxAccountId":"2","ctxStartEventId":"75"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2013-12-26 16:09:18,646 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Job-Executor-20:ctx-e25a8c36) Add job-24 into job monitoring
2013-12-26 16:09:18,646 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-20:ctx-e25a8c36) Executing AsyncJobVO {id:24, userId: 2,
accountId: 2, instanceType: Host, instanceId: 1, cmd:
org.apache.cloudstack.api.command.admin.host.PrepareForMaintenanceCmd, cmdInfo:
{"response":"json","id":"bd4e26d2-effb-4c2a-a22a-137a5d91225a","sessionkey":"d+syfd/Wn2mU+t9TnFJ4cyRpkbY\u003d","cmdEventType":"MAINT.PREPARE","ctxUserId":"2","httpmethod":"GET","_":"1388054354967","ctxAccountId":"2","ctxStartEventId":"75"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2013-12-26 16:09:18,647 DEBUG [c.c.a.ApiServlet] (catalina-exec-20:ctx-06e2a7d7
ctx-bf5e3753) ===END=== 10.146.0.134 -- GET
command=prepareHostForMaintenance&id=bd4e26d2-effb-4c2a-a22a-137a5d91225a&response=json&sessionkey=d%2Bsyfd%2FWn2mU%2Bt9TnFJ4cyRpkbY%3D&_=1388054354967
2013-12-26 16:09:18,660 DEBUG [c.c.a.t.Request] (Job-Executor-20:ctx-e25a8c36
ctx-bf5e3753) Seq 1-493617246: Sending { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] }
2013-12-26 16:09:18,660 DEBUG [c.c.a.t.Request] (Job-Executor-20:ctx-e25a8c36
ctx-bf5e3753) Seq 1-493617246: Executing: { Cmd , MgmtId: 132129494109518,
via: 1(10.147.40.14), Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] }
2013-12-26 16:09:18,660 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-110:ctx-691b6046) Seq 1-493617246: Executing request
2013-12-26 16:09:18,661 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-110:ctx-691b6046) POST request
tohttp://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.MaintainCommand
with contents{"contextMap":{},"wait":0}
2013-12-26 16:09:18,661 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-110:ctx-691b6046) Sending cmd to
http://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.MaintainCommand
cmd data:{"contextMap":{},"wait":0}
2013-12-26 16:09:18,669 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-110:ctx-691b6046) POST response
is[{"com.cloud.agent.api.MaintainAnswer":{"result":true,"details":"success -
NOP for MaintainCommand","_reconnect":false,"contextMap":{}}}]
2013-12-26 16:09:18,661 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-110:ctx-691b6046) Sending cmd to
http://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.MaintainCommand
cmd data:{"contextMap":{},"wait":0}
2013-12-26 16:09:18,669 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-110:ctx-691b6046) POST response
is[{"com.cloud.agent.api.MaintainAnswer":{"result":true,"details":"success -
NOP for MaintainCommand","_reconnect":false,"contextMap":{}}}]
2013-12-26 16:09:18,672 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-110:ctx-691b6046) executeRequest received response
[{"com.cloud.agent.api.MaintainAnswer":{"willMigrate":false,"result":true,"details":"success
- NOP for MaintainCommand","contextMap":{},"wait":0}}]
2013-12-26 16:09:18,672 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-110:ctx-691b6046) Seq 1-493617246: Response Received:
2013-12-26 16:09:18,673 DEBUG [c.c.a.t.Request] (DirectAgent-110:ctx-691b6046)
Seq 1-493617246: Processing: { Ans: , MgmtId: 132129494109518, via: 1, Ver:
v1, Flags: 110,
[{"com.cloud.agent.api.MaintainAnswer":{"willMigrate":false,"result":true,"details":"success
- NOP for MaintainCommand","wait":0}}] }
2013-12-26 16:09:18,673 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-110:ctx-691b6046) Seq 1-493617246: No more commands found
2013-12-26 16:09:18,673 DEBUG [c.c.a.t.Request] (Job-Executor-20:ctx-e25a8c36
ctx-bf5e3753) Seq 1-493617246: Received: { Ans: , MgmtId: 132129494109518,
via: 1, Ver: v1, Flags: 110, { MaintainAnswer } }
2013-12-26 16:09:18,673 DEBUG [c.c.a.m.AgentManagerImpl]
(Job-Executor-20:ctx-e25a8c36 ctx-bf5e3753) Details from executing class
com.cloud.agent.api.MaintainCommand: success - NOP for MaintainCommand
2013-12-26 16:09:18,877 DEBUG [c.c.r.ResourceState]
(Job-Executor-20:ctx-e25a8c36 ctx-bf5e3753) Resource state update: [id = 1;
name = 10.147.40.14; old state = Enabled; event = AdminAskMaintenace; new state
= PrepareForMaintenance]
2013-12-26 16:09:18,877 DEBUG [c.c.a.m.AgentAttache]
(Job-Executor-20:ctx-e25a8c36 ctx-bf5e3753) Seq 1-493617155: Sending disconnect
to class com.cloud.network.security.SecurityGroupListener
2013-12-26 16:09:18,937 DEBUG [c.c.h.HighAvailabilityManagerImpl]
(Job-Executor-20:ctx-e25a8c36 ctx-bf5e3753) Scheduled
HAWork[1-ForceStop-1-Running-Scheduled]
2013-12-26 16:09:18,986 INFO [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-6cc28d97 work-1) Processing
HAWork[1-ForceStop-1-Running-Scheduled]
2013-12-26 16:09:18,988 INFO [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-6cc28d97 work-1) Stopping VM[ConsoleProxy|v-1-VM]
2013-12-26 16:09:19,071 DEBUG [c.c.h.HighAvailabilityManagerImpl]
(Job-Executor-20:ctx-e25a8c36 ctx-bf5e3753) Scheduled
HAWork[2-ForceStop-2-Running-Scheduled]
2013-12-26 16:09:19,119 DEBUG [c.c.c.CapacityManagerImpl]
(HA-Worker-3:ctx-6cc28d97 work-1) VM state transitted from :Running to Stopping
with event: StopRequestedvm's original host id: 1 new host id: 1 host id before
state transition: 1
2013-12-26 16:09:19,123 DEBUG [c.c.a.t.Request] (HA-Worker-3:ctx-6cc28d97
work-1) Seq 1-493617247: Sending { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-1-VM","wait":0}}]
}
2013-12-26 16:09:19,124 DEBUG [c.c.a.t.Request] (HA-Worker-3:ctx-6cc28d97
work-1) Seq 1-493617247: Executing: { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-1-VM","wait":0}}]
}
2013-12-26 16:09:19,127 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-233:ctx-7d87c0a4) Seq 1-493617247: Executing request
2013-12-26 16:09:19,127 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-233:ctx-7d87c0a4) POST request
tohttp://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.StopCommand
with
contents{"isProxy":false,"executeInSequence":false,"vmName":"v-1-VM","contextMap":{},"wait":0}
2013-12-26 16:09:19,128 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-233:ctx-7d87c0a4) Sending cmd to
http://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.StopCommand cmd
data:{"isProxy":false,"executeInSequence":false,"vmName":"v-1-VM","contextMap":{},"wait":0}
2013-12-26 16:09:19,169 INFO [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-0:ctx-2adf59ff work-2) Processing
HAWork[2-ForceStop-2-Running-Scheduled]
2013-12-26 16:09:19,172 INFO [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-0:ctx-2adf59ff work-2) Stopping VM[SecondaryStorageVm|s-2-VM]
2013-12-26 16:09:19,246 DEBUG [c.c.h.HighAvailabilityManagerImpl]
(Job-Executor-20:ctx-e25a8c36 ctx-bf5e3753) Scheduled
HAWork[3-ForceStop-4-Running-Scheduled]
2013-12-26 16:09:19,303 DEBUG [c.c.c.CapacityManagerImpl]
(HA-Worker-0:ctx-2adf59ff work-2) VM state transitted from :Running to Stopping
with event: StopRequestedvm's original host id: 1 new host id: 1 host id before
state transition: 1
2013-12-26 16:09:19,306 DEBUG [c.c.a.t.Request] (HA-Worker-0:ctx-2adf59ff
work-2) Seq 1-493617248: Sending { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-2-VM","wait":0}}]
}
2013-12-26 16:09:19,306 DEBUG [c.c.a.t.Request] (HA-Worker-0:ctx-2adf59ff
work-2) Seq 1-493617248: Executing: { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-2-VM","wait":0}}]
}
2013-12-26 16:09:19,307 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-228:ctx-f2019a94) Seq 1-493617248: Executing request
2013-12-26 16:09:19,308 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-228:ctx-f2019a94) POST request
tohttp://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.StopCommand
with
contents{"isProxy":false,"executeInSequence":false,"vmName":"s-2-VM","contextMap":{},"wait":0}
2013-12-26 16:09:19,308 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-228:ctx-f2019a94) Sending cmd to
http://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.StopCommand cmd
data:{"isProxy":false,"executeInSequence":false,"vmName":"s-2-VM","contextMap":{},"wait":0}
2013-12-26 16:09:19,371 DEBUG [c.c.h.HighAvailabilityManagerImpl]
(Job-Executor-20:ctx-e25a8c36 ctx-bf5e3753) Scheduled
HAWork[4-ForceStop-5-Running-Scheduled]
2013-12-26 16:09:19,445 INFO [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-9b3f1e1e work-3) Processing
HAWork[3-ForceStop-4-Running-Scheduled]
2013-12-26 16:09:19,448 INFO [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-9b3f1e1e work-3) Stopping VM[DomainRouter|r-4-VM]
2013-12-26 16:09:19,596 INFO [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-17202997 work-4) Processing
HAWork[4-ForceStop-5-Running-Scheduled]
2013-12-26 16:09:19,598 INFO [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-17202997 work-4) Stopping VM[User|vm2]
2013-12-26 16:09:19,671 DEBUG [c.c.c.CapacityManagerImpl]
(HA-Worker-2:ctx-9b3f1e1e work-3) VM state transitted from :Running to Stopping
with event: StopRequestedvm's original host id: 1 new host id: 1 host id before
state transition: 1
2013-12-26 16:09:19,681 WARN [c.c.a.m.AgentManagerImpl]
(HA-Worker-2:ctx-9b3f1e1e work-3) Resource [Host:1] is unreachable: Host 1:
Unable to send class com.cloud.agent.api.NetworkUsageCommand because agent
10.147.40.14 is in maintenance mode
2013-12-26 16:09:19,686 DEBUG [c.c.a.t.Request] (HA-Worker-2:ctx-9b3f1e1e
work-3) Seq 1-493617250: Sending { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-4-VM","wait":0}}]
}
2013-12-26 16:09:19,686 DEBUG [c.c.a.t.Request] (HA-Worker-2:ctx-9b3f1e1e
work-3) Seq 1-493617250: Executing: { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-4-VM","wait":0}}]
}
2013-12-26 16:09:19,686 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-209:ctx-08009887) Seq 1-493617250: Executing request
2013-12-26 16:09:19,686 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-209:ctx-08009887) POST request
tohttp://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.StopCommand
with
contents{"isProxy":false,"executeInSequence":false,"vmName":"r-4-VM","contextMap":{},"wait":0}
2013-12-26 16:09:19,687 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-209:ctx-08009887) Sending cmd to
http://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.StopCommand cmd
data:{"isProxy":false,"executeInSequence":false,"vmName":"r-4-VM","contextMap":{},"wait":0}
2013-12-26 16:09:19,745 DEBUG [c.c.r.ResourceManagerImpl]
(Job-Executor-20:ctx-e25a8c36 ctx-bf5e3753) Sent resource event
EVENT_PREPARE_MAINTENANCE_AFTER to listener CapacityManagerImpl
2013-12-26 16:09:19,752 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-20:ctx-e25a8c36 ctx-bf5e3753) Complete async job-24, jobStatus:
SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.HostResponse/host/{"id":"bd4e26d2-effb-4c2a-a22a-137a5d91225a","name":"10.147.40.14","state":"Up","type":"Routing","ipaddress":"10.147.40.14","zoneid":"08a7fc58-ecaa-4919-b9ea-0e1b9b9457c6","zonename":"Adv","podid":"e113e9c7-dfa7-4c64-b6b7-4c10475de04a","podname":"pod1","version":"4.3.0-SNAPSHOT","hypervisor":"Hyperv","cpusockets":1,"cpunumber":4,"cpuspeed":2394,"cpuallocated":"0%","cpuused":"1%","cpuwithoverprovisioning":"9576.0","networkkbsread":0,"networkkbswrite":0,"memorytotal":17169604608,"memoryallocated":0,"memoryused":4460368,"capabilities":"hvm","lastpinged":"1970-01-16T22:01:59+0530","managementserverid":132129494109518,"clusterid":"febf7498-f7a9-4d49-9fb4-4d37fd1aa057","clustername":"Hyper-V","clustertype":"CloudManaged","islocalstorageactive":false,"created":"2013-12-26T15:28:27+0530","events":"AgentConnected;
PingTimeout; ShutdownRequested; Remove; ManagementServerDown; Ping;
AgentDisconnected; StartAgentRebalance;
HostDown","resourcestate":"PrepareForMaintenance","hypervisorversion":"6.2.9200.0","hahost":false,"jobid":"9b30d623-dfa0-4b09-ad06-b9859ce1b69c","jobstatus":0}
2013-12-26 16:09:19,813 DEBUG [c.c.c.CapacityManagerImpl]
(HA-Worker-4:ctx-17202997 work-4) VM state transitted from :Running to Stopping
with event: StopRequestedvm's original host id: 1 new host id: 1 host id before
state transition: 1
2013-12-26 16:09:19,813 DEBUG [c.c.c.CapacityManagerImpl]
(HA-Worker-4:ctx-17202997 work-4) VM state transitted from :Running to Stopping
with event: StopRequestedvm's original host id: 1 new host id: 1 host id before
state transition: 1
2013-12-26 16:09:19,818 DEBUG [c.c.a.t.Request] (HA-Worker-4:ctx-17202997
work-4) Seq 1-493617251: Sending { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-5-VM","wait":0}}]
}
2013-12-26 16:09:19,818 DEBUG [c.c.a.t.Request] (HA-Worker-4:ctx-17202997
work-4) Seq 1-493617251: Executing: { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-5-VM","wait":0}}]
}
2013-12-26 16:09:19,819 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-240:ctx-85423d2d) Seq 1-493617251: Executing request
2013-12-26 16:09:19,819 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-240:ctx-85423d2d) POST request
tohttp://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.StopCommand
with
contents{"isProxy":false,"executeInSequence":false,"vmName":"i-2-5-VM","contextMap":{},"wait":0}
2013-12-26 16:09:19,819 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-240:ctx-85423d2d) Sending cmd to
http://10.147.40.14:8250/api/HypervResource/com.cloud.agent.api.StopCommand cmd
data:{"isProxy":false,"executeInSequence":false,"vmName":"i-2-5-VM","contextMap":{},"wait":0}
2013-12-26 16:09:19,879 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-20:ctx-e25a8c36) Done executing
org.apache.cloudstack.api.command.admin.host.PrepareForMaintenanceCmd for job-24
2013-12-26 16:09:20,021 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Job-Executor-20:ctx-e25a8c36) Remove job-24 from job monitoring
2013-12-26 16:09:21,406 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-233:ctx-7d87c0a4) POST response
is[{"com.cloud.agent.api.StopAnswer":{"result":true,"details":null,"vm":null,"contextMap":{}}}]
mysql> select id,name,status,type,hypervisor_type,resource_state from host
where resource_state='Maintenance';
+----+--------------+--------+---------+-----------------+----------------+
| id | name | status | type | hypervisor_type | resource_state |
+----+--------------+--------+---------+-----------------+----------------+
| 1 | 10.147.40.14 | Up | Routing | Hyperv | Maintenance |
+----+--------------+--------+---------+-----------------+----------------+
mysql> select id,name,instance_name,state from vm_instance where
last_host_id=1 and host_id is NULL;
+----+--------+---------------+---------+
| id | name | instance_name | state |
+----+--------+---------------+---------+
| 4 | r-4-VM | r-4-VM | Stopped |
| 5 | vm2 | i-2-5-VM | Stopped |
+----+--------+---------------+---------+
2 rows in set (0.00 sec)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
