Hi Madhu,
Please find requested info.
BR
Salam
From: Madhu Rajanna <[email protected]>
To: Shaik Salam <[email protected]>
Cc: "[email protected] List" <[email protected]>,
Michael Adam <[email protected]>
Date: 01/24/2019 01:33 PM
Subject: Re: Failed to provision volume with StorageClass
"glusterfs-storage": glusterfs: server busy
"External email. Open with Caution"
the heketi logs you have attached is not complete i believe, can you
povide the complete heketi logs
and also an we get the output of "ps aux" from the gluster pods ? I want
to see if any lvm commands or gluster commands are "stuck".
On Thu, Jan 24, 2019 at 1:16 PM Shaik Salam <[email protected]> wrote:
Hi Madhu.
I tried lot of times restarted heketi pod but not resolved.
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 0
New: 0
Stale: 0
Now you can see all operations are zero. Now I try to create single volume
below is observation in-flight reaching slowly to 8.
sh-4.4# heketi-cli server operations infoCLI_SERVER=http://localhost:8080
; export HEKETI_CLI_USE Operation
Counts:
Total: 0
In-Flight: 6
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
[negroni] Completed 200 OK in 186.286µs
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 166.294µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 186.411µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 179.796µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 131.108µs
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 111.392µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 265.023µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 179.364µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 295.058µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 146.857µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/24 07:43:36 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 403.166µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/24 07:43:51 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 193.554µs
But for pod volume is not creating.
1:15:36 PM
Warning
Provisioning failed
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume Server busy. Retry
operation later..
9 times in the last 2 minutes
1:13:21 PM
Warning
Provisioning failed
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume .
8 times in the last
From: "Madhu Rajanna" <[email protected]>
To: "Shaik Salam" <[email protected]>
Cc: "[email protected] List" <[email protected]>,
"Michael Adam" <[email protected]>
Date: 01/24/2019 12:51 PM
Subject: Re: Failed to provision volume with StorageClass
"glusterfs-storage": glusterfs: server busy
"External email. Open with Caution"
HI Shaik,
can you provide me the outpout of $heketi-cli server operations info
from heketi pod
as a workround you can try restarting the heketi pod. This will cause the
current operations to go stale, but other pending pvcs may go to Bound
state
Regards,
Madhu R
On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <[email protected]> wrote:
H Madhu,
Could you please have look my issue If you have time (atleast workaround).
I am unable to send mail to "John Mulligan" <[email protected]>"
who is currently handling issue
https://bugzilla.redhat.com/show_bug.cgi?id=1636912
BR
Salam
From: Shaik Salam/HYD/TCS
To: "John Mulligan" <[email protected]>, "Michael Adam" <
[email protected]>, "Madhu Rajanna" <[email protected]>
Cc: "[email protected] List" <[email protected]>
Date: 01/24/2019 12:21 PM
Subject: Failed to provision volume with StorageClass
"glusterfs-storage": glusterfs: server busy
Hi All,
We are facing also following issue on openshift origin while we are
creating pvc for pods. (atlease provide workaround to move further)
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume Server busy. Retry
operation later..
Please find heketidb dump and log
[negroni] Completed 429 Too Many Requests in 250.763µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 169.08µs
[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c
[negroni] Completed 404 Not Found in 148.125µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 496.624µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 101.673µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 209.681µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 103.595µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 297.594µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 96.75µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 477.007µs
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 165.38µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 488.253µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 171.836µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 208.59µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds
limit (8)
[negroni] Completed 429 Too Many Requests in 125.141µs
[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606
[negroni] Completed 404 Not Found in 138.687µs
[negroni] Started POST /volumes
BR
Salam
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
--
Madhu Rajanna
Software Engineer
Red Hat Bangalore, India
[email protected] M: +91-9741133155
--
Madhu Rajanna
Software Engineer
Red Hat Bangalore, India
[email protected] M: +91-9741133155
[root@master ~]# oc get po
NAME READY STATUS RESTARTS AGE
glusterblock-storage-provisioner-dc-2-b46r0 1/1 Running 0 7d
glusterfs-storage-31fjz 1/1 Running 0
104d
glusterfs-storage-9h5cf 1/1 Running 0
104d
glusterfs-storage-pnb67 1/1 Running 0
104d
heketi-storage-6-4m61c 1/1 Running 0 36m
[root@master ~]# oc rsh glusterfs-storage-31fjz
sh-4.2# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 44276 4248 ? Ss 2018 18:59 /usr/sbin/init
root 29 0.0 0.0 109848 57912 ? Ss 2018 3:06 /usr/lib/syste
root 34 0.0 0.0 1706204 163716 ? SLsl 2018 77:03 /usr/sbin/dmev
root 40 0.0 0.0 26096 1588 ? Ss 2018 0:15 /usr/sbin/cron
root 63 0.0 0.0 389612 4116 ? Ssl 2018 0:00 /usr/bin/tcmu-
root 64 0.0 0.0 94900 968 ? Ssl 2018 0:00 /usr/sbin/gssp
rpc 65 0.0 0.0 69224 1048 ? Ss 2018 0:12 /sbin/rpcbind
dbus 75 0.0 0.0 58068 2044 ? Ss 2018 0:13 /usr/bin/dbus-
root 92 0.0 0.0 112816 2296 ? Ss 2018 0:00 /usr/sbin/sshd
root 133 0.0 0.0 16544 2972 ? Ss 2018 1:43 /bin/bash /usr
root 245 0.0 0.0 26380 1664 ? Ss 2018 0:24 /usr/lib/syste
root 576 0.0 0.0 1423140 29968 ? Ssl 2018 35:46 /usr/sbin/glus
root 869 0.0 0.0 1660308 44588 ? Ssl 2018 43:11 /usr/sbin/glus
root 1934 0.0 0.0 2199692 98344 ? Ssl 2018 85:34 /usr/sbin/glus
root 3907 0.0 0.0 2197092 48768 ? Ssl 2018 73:40 /usr/sbin/glus
root 4486 0.0 0.0 2200992 56884 ? Ssl 2018 131:30 /usr/sbin/glus
root 6510 20.2 0.3 27866580 925756 ? Ssl 2018 29338:24 /usr/sbin/gl
root 11149 1.1 0.0 2265488 58244 ? Ssl 2018 1694:33 /usr/sbin/glu
root 15336 0.0 0.0 11820 1788 pts/7 Ss+ Jan10 0:00 /bin/sh
root 16908 0.0 0.0 1210152 18616 ? Ssl 2018 21:22 /usr/sbin/glus
root 23994 0.0 0.0 11820 1732 pts/10 Ss+ Jan10 0:00 /bin/sh
root 33780 0.3 0.0 574992 20348 ? Ssl Jan18 26:57 /usr/sbin/glus
root 33821 0.0 0.0 202380 3752 ? Ssl Jan18 0:00 /usr/sbin/glus
root 33863 0.0 0.0 2096820 59940 ? Ssl Jan18 0:35 /usr/sbin/glus
root 42784 0.0 0.0 11820 1448 pts/12 Ss+ Jan17 0:00 /bin/sh
root 54573 0.0 0.0 11820 1492 pts/6 Ss+ 2018 0:00 /bin/sh
root 73119 0.0 0.0 15252 1988 pts/19 Ss+ 05:29 0:00 bash
root 73126 0.0 0.0 1607008 27416 ? Ssl 2018 66:02 /usr/sbin/glus
root 76135 0.0 0.0 1210152 18572 ? Ssl 2018 31:22 /usr/sbin/glus
root 76350 0.0 0.0 11820 1648 pts/11 Ss+ 2018 0:00 /bin/sh
root 86701 0.0 0.0 7768 348 ? S 08:06 0:00 sleep 120
root 86797 0.0 0.0 11820 1716 pts/14 Ss 08:07 0:00 /bin/sh
root 86831 0.0 0.0 51712 1724 pts/14 R+ 08:07 0:00 ps aux
root 87644 0.0 0.0 1473336 31268 ? Ssl 2018 29:08 /usr/sbin/glus
root 90792 0.0 0.0 11820 1472 pts/13 Ss+ Jan22 0:00 /bin/sh
root 91110 0.2 0.0 1671764 31624 ? Ssl 2018 355:14 /usr/sbin/glus
root 92244 0.2 0.0 1539652 26904 ? Ssl 2018 393:05 /usr/sbin/glus
root 103360 0.0 0.0 1510760 46724 ? Ssl 2018 26:43 /usr/sbin/glus
root 109344 0.0 0.0 1935696 44764 ? Ssl 2018 121:16 /usr/sbin/glus
root 109591 0.0 0.0 1933648 37484 ? Ssl 2018 72:41 /usr/sbin/glus
root 109834 0.0 0.0 1802056 38212 ? Ssl 2018 83:06 /usr/sbin/glus
root 121593 0.0 0.0 11820 1524 pts/0 Ss+ 2018 0:00 /bin/sh
root 123388 0.0 0.0 1210152 20244 ? Ssl 2018 35:39 /usr/sbin/glus
sh-4.2# exit
exit
[root@master ~]# oc rsh glusterfs-storage-9h5cf
sh-4.2# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 43760 3948 ? Ss 2018 17:43 /usr/sbin/init
root 29 0.0 0.0 85372 32048 ? Ss 2018 0:56 /usr/lib/syste
root 30 0.0 0.0 1706204 162496 ? SLsl 2018 98:00 /usr/sbin/dmev
root 41 0.0 0.0 26100 1596 ? Ss 2018 0:11 /usr/sbin/cron
rpc 56 0.0 0.0 69224 1128 ? Ss 2018 0:10 /sbin/rpcbind
root 63 0.0 0.0 94900 1232 ? Ssl 2018 0:00 /usr/sbin/gssp
root 70 0.0 0.0 389612 7996 ? Ssl 2018 0:00 /usr/bin/tcmu-
dbus 75 0.0 0.0 58068 2316 ? Ss 2018 0:10 /usr/bin/dbus-
root 92 0.0 0.0 112816 4296 ? Ss 2018 0:00 /usr/sbin/sshd
root 132 0.0 0.0 16540 2984 ? Ss 2018 1:44 /bin/bash /usr
root 245 0.0 0.0 26456 1804 ? Ss 2018 0:21 /usr/lib/syste
root 558 0.0 0.0 1493876 27780 ? Ssl 2018 20:21 /usr/sbin/glus
root 901 0.0 0.0 1676948 51548 ? Ssl 2018 28:58 /usr/sbin/glus
root 2130 0.0 0.0 2197092 44080 ? Ssl 2018 62:59 /usr/sbin/glus
root 2713 0.0 0.0 2200732 55248 ? Ssl 2018 140:44 /usr/sbin/glus
root 3586 0.0 0.0 7768 344 ? S 08:08 0:00 sleep 120
root 3604 0.3 0.0 11820 1708 pts/6 Ss 08:08 0:00 /bin/sh
root 3613 0.0 0.0 51712 1720 pts/6 R+ 08:08 0:00 ps aux
root 9064 1.1 0.0 2000744 60192 ? Ssl 2018 1683:41 /usr/sbin/glu
root 23952 0.4 0.0 574992 13984 ? Ssl Jan18 38:53 /usr/sbin/glus
root 23995 0.0 0.0 202380 5812 ? Ssl Jan18 0:00 /usr/sbin/glus
root 24048 0.0 0.0 2096820 59372 ? Ssl Jan18 0:38 /usr/sbin/glus
root 58228 0.0 0.0 11820 1692 pts/3 Ss+ 2018 0:00 /bin/sh
root 69135 0.0 0.0 11820 1696 pts/4 Ss+ Jan18 0:00 /bin/sh
root 70604 0.0 0.0 15252 2008 pts/1 Ss+ 2018 0:00 /bin/bash
root 71770 0.0 0.0 1210152 20144 ? Ssl 2018 19:25 /usr/sbin/glus
root 72776 0.0 0.0 1540692 32344 ? Ssl 2018 55:05 /usr/sbin/glus
root 73279 0.0 0.0 15252 1976 pts/2 Ss+ 2018 0:00 /bin/bash
root 82920 0.0 0.0 1407540 28548 ? Ssl 2018 19:29 /usr/sbin/glus
root 86318 0.2 0.0 1737300 33392 ? Ssl 2018 374:55 /usr/sbin/glus
root 87342 0.2 0.0 1539652 25968 ? Ssl 2018 345:17 /usr/sbin/glus
root 97945 0.0 0.0 1469144 52052 ? Ssl 2018 20:23 /usr/sbin/glus
root 98706 0.0 0.0 11820 1704 pts/5 Ss+ Jan20 0:00 /bin/sh
root 103617 0.0 0.0 1867852 46592 ? Ssl 2018 82:56 /usr/sbin/glus
root 103808 0.0 0.0 1933648 39320 ? Ssl 2018 77:32 /usr/sbin/glus
root 103990 0.0 0.0 1802056 40996 ? Ssl 2018 88:52 /usr/sbin/glus
root 105270 0.0 0.0 1210152 18388 ? Ssl 2018 18:12 /usr/sbin/glus
root 120069 0.0 0.0 11820 1688 pts/0 Ss+ 2018 0:00 /bin/sh
root 121771 0.0 0.0 1210152 21780 ? Ssl 2018 19:35 /usr/sbin/glus
root 130924 0.0 0.0 2198912 95808 ? Ssl 2018 78:58 /usr/sbin/glus
sh-4.2# exit
exit
[root@master ~]# oc rsh glusterfs-storage-pnb67
sh-4.2# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 44040 4012 ? Ss 2018 13:00 /usr/sbin/init
root 29 0.0 0.0 72864 25956 ? Ss 2018 6:23 /usr/lib/syste
root 33 0.0 0.0 1442828 160844 ? SLsl 2018 55:54 /usr/sbin/dmev
root 48 0.0 0.0 26104 1600 ? Ss 2018 0:16 /usr/sbin/cron
root 63 0.0 0.0 389612 6716 ? Ssl 2018 0:00 /usr/bin/tcmu-
rpc 64 0.0 0.0 69224 1100 ? Ss 2018 0:13 /sbin/rpcbind
root 65 0.0 0.0 94900 1092 ? Ssl 2018 0:00 /usr/sbin/gssp
dbus 75 0.0 0.0 58068 2088 ? Ss 2018 0:56 /usr/bin/dbus-
root 92 0.0 0.0 112816 3300 ? Ss 2018 0:00 /usr/sbin/sshd
root 134 0.0 0.0 16540 2976 ? Ss 2018 1:41 /bin/bash /usr
root 245 0.0 0.0 26488 1712 ? Ss 2018 0:33 /usr/lib/syste
root 547 0.0 0.0 1357604 29068 ? Ssl 2018 24:59 /usr/sbin/glus
root 861 0.0 0.0 1931520 51168 ? Ssl 2018 31:42 /usr/sbin/glus
root 1980 0.0 0.0 2197092 45064 ? Ssl 2018 59:36 /usr/sbin/glus
root 2552 0.0 0.0 2199432 52012 ? Ssl 2018 129:05 /usr/sbin/glus
root 8809 1.0 0.0 2265228 59684 ? Ssl 2018 1516:13 /usr/sbin/glu
root 24708 0.0 0.0 1210152 18020 ? Ssl 2018 21:51 /usr/sbin/glus
root 47365 0.0 0.0 7768 348 ? S 08:07 0:00 sleep 120
root 47404 0.0 0.0 11820 1704 pts/11 Ss 08:08 0:00 /bin/sh
root 47411 0.0 0.0 51712 1724 pts/11 R+ 08:08 0:00 ps aux
root 70656 0.0 0.0 1210152 19764 ? Ssl 2018 22:56 /usr/sbin/glus
root 73468 0.0 0.0 1540692 29904 ? Ssl 2018 54:21 /usr/sbin/glus
root 75649 0.0 0.0 11820 1812 pts/6 Ss+ Jan21 0:00 /bin/sh
root 81683 0.0 0.0 1407540 32992 ? Ssl 2018 23:40 /usr/sbin/glus
root 85052 0.2 0.0 2066020 33628 ? Ssl 2018 358:31 /usr/sbin/glus
root 86073 0.2 0.0 1605708 20324 ? Ssl 2018 300:28 /usr/sbin/glus
root 97021 0.0 0.0 1570056 43760 ? Ssl 2018 24:23 /usr/sbin/glus
root 98697 0.3 0.0 509456 13748 ? Ssl Jan21 14:41 /usr/sbin/glus
root 98741 0.0 0.0 202380 1724 ? Ssl Jan21 0:00 /usr/sbin/glus
root 98786 0.0 0.0 2031284 36664 ? Ssl Jan21 0:18 /usr/sbin/glus
root 101995 0.0 0.0 11820 1704 pts/2 Ss+ Jan21 0:00 /bin/sh
root 103063 0.0 0.0 1999444 41524 ? Ssl 2018 77:04 /usr/sbin/glus
root 103276 0.0 0.0 1867852 48060 ? Ssl 2018 116:27 /usr/sbin/glus
root 103457 0.0 0.0 1802056 46200 ? Ssl 2018 120:07 /usr/sbin/glus
root 109041 0.0 0.0 15252 1972 pts/1 Ss+ 2018 0:00 /bin/bash
root 120197 0.0 0.0 11820 1680 pts/3 Ss+ 2018 0:00 /bin/sh
root 121840 0.0 0.0 1210152 21320 ? Ssl 2018 24:05 /usr/sbin/glus
root 130810 0.0 0.0 2200732 92676 ? Ssl 2018 76:51 /usr/sbin/glus
_______________________________________________
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users
