[ovirt-users] vdsm command failed connection reset by peer

2015-12-12 Thread alireza sadeh seighalan 
hi everyone


today i saw an error in events tab of ovirt3.6. it shows me this error:

VDSM HV07   command failed connection reset by peer

my vdsm and libvirt services status is:


[root@hv07 ~]# systemctl status vdsmd
vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
   Active: active (running) since Thu 2015-12-10 17:28:10 IRST; 2 days ago
  Process: 17321 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
--post-stop (code=exited, status=0/SUCCESS)
  Process: 17324 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
 Main PID: 17417 (vdsm)
   CGroup: /system.slice/vdsmd.service
   ├─ 2407 /usr/libexec/ioprocess --read-pipe-fd 151
--write-pipe-fd 147 --max-threads 10 --max-queued-requests 10
   ├─ 4072 /usr/libexec/ioprocess --read-pipe-fd 159
--write-pipe-fd 156 --max-threads 10 --max-queued-requests 10
   ├─ 4190 /usr/libexec/ioprocess --read-pipe-fd 183
--write-pipe-fd 180 --max-threads 10 --max-queued-requests 10
   ├─ 4212 /usr/libexec/ioprocess --read-pipe-fd 153
--write-pipe-fd 150 --max-threads 10 --max-queued-requests 10
   ├─ 4234 /usr/libexec/ioprocess --read-pipe-fd 49 --write-pipe-fd
46 --max-threads 10 --max-queued-requests 10
   ├─14269 /usr/libexec/ioprocess --read-pipe-fd 35 --write-pipe-fd
34 --max-threads 10 --max-queued-requests 10
   ├─14280 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd
45 --max-threads 10 --max-queued-requests 10
   ├─14289 /usr/libexec/ioprocess --read-pipe-fd 56 --write-pipe-fd
55 --max-threads 10 --max-queued-requests 10
   ├─14296 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd
64 --max-threads 10 --max-queued-requests 10
   ├─14303 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd
72 --max-threads 10 --max-queued-requests 10
   ├─14310 /usr/libexec/ioprocess --read-pipe-fd 82 --write-pipe-fd
80 --max-threads 10 --max-queued-requests 10
   ├─14317 /usr/libexec/ioprocess --read-pipe-fd 90 --write-pipe-fd
89 --max-threads 10 --max-queued-requests 10
   ├─14324 /usr/libexec/ioprocess --read-pipe-fd 97 --write-pipe-fd
96 --max-threads 10 --max-queued-requests 10
   ├─14331 /usr/libexec/ioprocess --read-pipe-fd 105
--write-pipe-fd 104 --max-threads 10 --max-queued-requests 10
   ├─14338 /usr/libexec/ioprocess --read-pipe-fd 117
--write-pipe-fd 113 --max-threads 10 --max-queued-requests 10
   ├─17417 /usr/bin/python /usr/share/vdsm/vdsm
   ├─17692 /usr/libexec/ioprocess --read-pipe-fd 131
--write-pipe-fd 130 --max-threads 10 --max-queued-requests 10
   ├─21151 /usr/libexec/ioprocess --read-pipe-fd 126
--write-pipe-fd 124 --max-threads 10 --max-queued-requests 10
   ├─21215 /usr/libexec/ioprocess --read-pipe-fd 140
--write-pipe-fd 139 --max-threads 10 --max-queued-requests 10
   ├─21446 /usr/libexec/ioprocess --read-pipe-fd 170
--write-pipe-fd 169 --max-threads 10 --max-queued-requests 10
   └─21511 /usr/libexec/ioprocess --read-pipe-fd 178
--write-pipe-fd 177 --max-threads 10 --max-queued-requests 10

Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(589) Request queue full
Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(589) Request queue full
Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(590) Request queue full
Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(590) Request queue full
Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(591) Request queue full
Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(591) Request queue full
Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(592) Request queue full
Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(592) Request queue full
Dec 11 16:54:07 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(593) Request queue full
Dec 11 16:54:10 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
(594) Request queue full


[root@hv07 ~]# systemctl status libvirtd
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
  Drop-In: /etc/systemd/system/libvirtd.service.d
   └─unlimited-core.conf
   Active: active (running) since Tue 2015-12-01 09:11:30 IRST; 1 weeks 4
days ago
 Docs: man:libvirtd(8)
   http://libvirt.org
 Main PID: 4523 (libvirtd)
   CGroup: /system.slice/libvirtd.service
   └─4523 /usr/sbin/libvirtd --listen

Dec 01 09:11:29 hv07.election.local libvirtd[4523]: libvirt version: 1.2.8,
package: 16.el7_1.5 (CentOS BuildSystem

Re: [ovirt-users] vdsm command failed connection reset by peer

2015-12-12 Thread Nir Soffer 
On Sat, Dec 12, 2015 at 7:17 PM, alireza sadeh seighalan
 wrote:
> hi everyone
>
>
> today i saw an error in events tab of ovirt3.6. it shows me this error:
>
> VDSM HV07   command failed connection reset by peer
>
> my vdsm and libvirt services status is:
>
>
> [root@hv07 ~]# systemctl status vdsmd
> vdsmd.service - Virtual Desktop Server Manager
>Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
>Active: active (running) since Thu 2015-12-10 17:28:10 IRST; 2 days ago
>   Process: 17321 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
> --post-stop (code=exited, status=0/SUCCESS)
>   Process: 17324 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
> --pre-start (code=exited, status=0/SUCCESS)
>  Main PID: 17417 (vdsm)
>CGroup: /system.slice/vdsmd.service
>├─ 2407 /usr/libexec/ioprocess --read-pipe-fd 151 --write-pipe-fd
> 147 --max-threads 10 --max-queued-requests 10
>├─ 4072 /usr/libexec/ioprocess --read-pipe-fd 159 --write-pipe-fd
> 156 --max-threads 10 --max-queued-requests 10
>├─ 4190 /usr/libexec/ioprocess --read-pipe-fd 183 --write-pipe-fd
> 180 --max-threads 10 --max-queued-requests 10
>├─ 4212 /usr/libexec/ioprocess --read-pipe-fd 153 --write-pipe-fd
> 150 --max-threads 10 --max-queued-requests 10
>├─ 4234 /usr/libexec/ioprocess --read-pipe-fd 49 --write-pipe-fd
> 46 --max-threads 10 --max-queued-requests 10
>├─14269 /usr/libexec/ioprocess --read-pipe-fd 35 --write-pipe-fd
> 34 --max-threads 10 --max-queued-requests 10
>├─14280 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd
> 45 --max-threads 10 --max-queued-requests 10
>├─14289 /usr/libexec/ioprocess --read-pipe-fd 56 --write-pipe-fd
> 55 --max-threads 10 --max-queued-requests 10
>├─14296 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd
> 64 --max-threads 10 --max-queued-requests 10
>├─14303 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd
> 72 --max-threads 10 --max-queued-requests 10
>├─14310 /usr/libexec/ioprocess --read-pipe-fd 82 --write-pipe-fd
> 80 --max-threads 10 --max-queued-requests 10
>├─14317 /usr/libexec/ioprocess --read-pipe-fd 90 --write-pipe-fd
> 89 --max-threads 10 --max-queued-requests 10
>├─14324 /usr/libexec/ioprocess --read-pipe-fd 97 --write-pipe-fd
> 96 --max-threads 10 --max-queued-requests 10
>├─14331 /usr/libexec/ioprocess --read-pipe-fd 105 --write-pipe-fd
> 104 --max-threads 10 --max-queued-requests 10
>├─14338 /usr/libexec/ioprocess --read-pipe-fd 117 --write-pipe-fd
> 113 --max-threads 10 --max-queued-requests 10
>├─17417 /usr/bin/python /usr/share/vdsm/vdsm
>├─17692 /usr/libexec/ioprocess --read-pipe-fd 131 --write-pipe-fd
> 130 --max-threads 10 --max-queued-requests 10
>├─21151 /usr/libexec/ioprocess --read-pipe-fd 126 --write-pipe-fd
> 124 --max-threads 10 --max-queued-requests 10
>├─21215 /usr/libexec/ioprocess --read-pipe-fd 140 --write-pipe-fd
> 139 --max-threads 10 --max-queued-requests 10
>├─21446 /usr/libexec/ioprocess --read-pipe-fd 170 --write-pipe-fd
> 169 --max-threads 10 --max-queued-requests 10
>└─21511 /usr/libexec/ioprocess --read-pipe-fd 178 --write-pipe-fd
> 177 --max-threads 10 --max-queued-requests 10
>
> Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (589) Request queue full
> Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (589) Request queue full
> Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (590) Request queue full
> Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (590) Request queue full
> Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (591) Request queue full
> Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (591) Request queue full
> Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (592) Request queue full
> Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (592) Request queue full
> Dec 11 16:54:07 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (593) Request queue full
> Dec 11 16:54:10 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (594) Request queue full

Looks like some of your storage is very slow or not accessible, so
IOProcess request queue becomes full.

To understand this issue, we need vdsm and engine logs. The info
available provided by systemctl not enough.

Please open a bug and attach both vdsm.log and engine.log showing the
time when you had this issue.

Adding Yeela.

Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users