Adding John who is having more idea about how to debug this one. @Shaik Salam <[email protected]> can you some more info on the hardware on which you are running heketi (kernel details)
On Thu, Jan 24, 2019 at 7:42 PM Shaik Salam <[email protected]> wrote: > Hi Madhu, > > Sorry to disturb could you please provide atleast work around (to clear > requests which stuck) to move further. > We are also not able to find root cause from glusterd logs. Please find > attachment. > > BR > Salam > > > > > > From: Shaik Salam/HYD/TCS > To: "Madhu Rajanna" <[email protected]> > Cc: "[email protected] List" <[email protected]>, > "Michael Adam" <[email protected]> > Date: 01/24/2019 04:12 PM > Subject: Re: Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: server busy > ------------------------------ > > > Hi Madhu, > > Please let me know If any other information required. > > BR > Salam > > > > > From: Shaik Salam/HYD/TCS > To: "Madhu Rajanna" <[email protected]> > Cc: "[email protected] List" <[email protected]>, > "Michael Adam" <[email protected]> > Date: 01/24/2019 03:23 PM > Subject: Re: Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: server busy > ------------------------------ > > > Hi Madhu, > > This is complete one after restart of heketi pod and process log. > > BR > Salam > > [attachment "heketi-pod-complete.log" deleted by Shaik Salam/HYD/TCS] > [attachment "ps-aux.txt" deleted by Shaik Salam/HYD/TCS] > > > > > From: "Madhu Rajanna" <[email protected]> > To: "Shaik Salam" <[email protected]> > Cc: "[email protected] List" <[email protected]>, > "Michael Adam" <[email protected]> > Date: 01/24/2019 01:55 PM > Subject: Re: Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: server busy > ------------------------------ > > > > *"External email. Open with Caution"* > the logs you provided is not complete, not able to figure out which > command is struck, can you reattach the complete output of `ps aux` and > also attach complete heketi logs. > > On Thu, Jan 24, 2019 at 1:41 PM Shaik Salam <*[email protected]* > <[email protected]>> wrote: > Hi Madhu, > > Please find requested info. > > BR > Salam > > > > > > From: Madhu Rajanna <*[email protected]* <[email protected]>> > To: Shaik Salam <*[email protected]* <[email protected]>> > Cc: "*[email protected]* <[email protected]> List" > <*[email protected]* <[email protected]>>, Michael Adam < > *[email protected]* <[email protected]>> > Date: 01/24/2019 01:33 PM > Subject: Re: Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: server busy > ------------------------------ > > > > * "External email. Open with Caution"* > the heketi logs you have attached is not complete i believe, can you > povide the complete heketi logs > and also an we get the output of "ps aux" from the gluster pods ? I want > to see if any lvm commands or gluster commands are "stuck". > > > On Thu, Jan 24, 2019 at 1:16 PM Shaik Salam <*[email protected]* > <[email protected]>> wrote: > Hi Madhu. > > I tried lot of times restarted heketi pod but not resolved. > > sh-4.4# heketi-cli server operations info > Operation Counts: > Total: 0 > In-Flight: 0 > New: 0 > Stale: 0 > > Now you can see all operations are zero. Now I try to create single volume > below is observation in-flight reaching slowly to 8. > > sh-4.4# heketi-cli server operations infoCLI_SERVER= > *http://localhost:8080* <http://localhost:8080/> ; export HEKETI_CLI_USE > Operation Counts: > Total: 0 > In-Flight: 6 > New: 0 > Stale: 0 > sh-4.4# heketi-cli server operations info > Operation Counts: > Total: 0 > In-Flight: 7 > New: 0 > Stale: 0 > sh-4.4# heketi-cli server operations info > Operation Counts: > Total: 0 > In-Flight: 7 > New: 0 > Stale: 0 > sh-4.4# heketi-cli server operations info > Operation Counts: > Total: 0 > In-Flight: 7 > New: 0 > Stale: 0 > sh-4.4# heketi-cli server operations info > Operation Counts: > Total: 0 > In-Flight: 7 > New: 0 > Stale: 0 > > [negroni] Completed 200 OK in 186.286µs > [negroni] Started POST /volumes > [negroni] Started GET /operations > [negroni] Completed 200 OK in 166.294µs > [negroni] Started GET /operations > [negroni] Completed 200 OK in 186.411µs > [negroni] Started GET /operations > [negroni] Completed 200 OK in 179.796µs > [negroni] Started POST /volumes > [negroni] Started POST /volumes > [negroni] Started POST /volumes > [negroni] Started POST /volumes > [negroni] Started GET /operations > [negroni] Completed 200 OK in 131.108µs > [negroni] Started POST /volumes > [negroni] Started GET /operations > [negroni] Completed 200 OK in 111.392µs > [negroni] Started GET /operations > [negroni] Completed 200 OK in 265.023µs > [negroni] Started GET /operations > [negroni] Completed 200 OK in 179.364µs > [negroni] Started GET /operations > [negroni] Completed 200 OK in 295.058µs > [negroni] Started GET /operations > [negroni] Completed 200 OK in 146.857µs > [negroni] Started POST /volumes > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/24 07:43:36 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 403.166µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/24 07:43:51 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 193.554µs > > > But for pod volume is not creating. > 1:15:36 PM > Warning > Provisioning failed Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: create volume err: error creating volume > Server busy. Retry operation later.. > 9 times in the last 2 minutes > 1:13:21 PM > Warning > Provisioning failed Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: create volume err: error creating volume . > 8 times in the last > > > > > > > From: "Madhu Rajanna" <*[email protected]* <[email protected]>> > To: "Shaik Salam" <*[email protected]* <[email protected]>> > Cc: "*[email protected]* <[email protected]> List" > <*[email protected]* <[email protected]>>, "Michael Adam" > <*[email protected]* <[email protected]>> > Date: 01/24/2019 12:51 PM > Subject: Re: Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: server busy > ------------------------------ > > > > * "External email. Open with Caution"* > HI Shaik, > > can you provide me the outpout of $heketi-cli server operations info > from heketi pod > > as a workround you can try restarting the heketi pod. This will cause the > current operations to go stale, but other pending pvcs may go to Bound > state > > Regards, > > Madhu R > > On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <*[email protected]* > <[email protected]>> wrote: > H Madhu, > > Could you please have look my issue If you have time (atleast workaround). > I am unable to send mail to "John Mulligan" <*[email protected]* > <[email protected]>>" who is currently handling issue > *https://bugzilla.redhat.com/show_bug.cgi?id=1636912* > <https://bugzilla.redhat.com/show_bug.cgi?id=1636912> > > BR > Salam > > > From: Shaik Salam/HYD/TCS > To: "John Mulligan" <*[email protected]* > <[email protected]>>, "Michael Adam" <*[email protected]* > <[email protected]>>, "Madhu Rajanna" <*[email protected]* > <[email protected]>> > Cc: "*[email protected]* <[email protected]> List" > <*[email protected]* <[email protected]>> > Date: 01/24/2019 12:21 PM > Subject: Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: server busy > > ------------------------------ > > > > > Hi All, > > We are facing also following issue on openshift origin while we are > creating pvc for pods. (atlease provide workaround to move further) > > Failed to provision volume with StorageClass "glusterfs-storage": > glusterfs: create volume err: error creating volume > Failed to provision volume with StorageClass "glusterfs-storage": > glusterfs: create volume err: error creating volume Server busy. Retry > operation later.. > > Please find heketidb dump and log > > [negroni] Completed 429 Too Many Requests in 250.763µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 169.08µs > [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c > [negroni] Completed 404 Not Found in 148.125µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 496.624µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 101.673µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 209.681µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 103.595µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 297.594µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 96.75µs > [negroni] Started POST /volumes > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 477.007µs > [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 165.38µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 488.253µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 171.836µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 208.59µs > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 125.141µs > [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 > [negroni] Completed 404 Not Found in 138.687µs > [negroni] Started POST /volumes > > > BR > Salam > =====-----=====-----===== > Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > > > > -- > Madhu Rajanna > Software Engineer > *Red Hat Bangalore, India* <https://www.redhat.com/> > *[email protected]* <[email protected]> M: +91-9741133155 > > <https://red.ht/sig> > > > > -- > Madhu Rajanna > Software Engineer > *Red Hat Bangalore, India* <https://www.redhat.com/> > *[email protected]* <[email protected]> M: +91-9741133155 > > <https://red.ht/sig> > > > > -- > Madhu Rajanna > Software Engineer > Red Hat Bangalore, India <https://www.redhat.com/> > [email protected] M: +91-9741133155 > > <https://red.ht/sig> > > -- Madhu Rajanna Software Engineer Red Hat Bangalore, India <https://www.redhat.com> [email protected] M: +91-9741133155 <https://red.ht/sig>
_______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
