I forgot to add that I also restarted three GVDs that were interrupted by the outage: - https://gerrit.cloudera.org/#/c/17314 (originally started by Joe for John Sherman) - https://gerrit.cloudera.org/#/c/17252 (originally started by Joe for Qifan) - https://gerrit.cloudera.org/#/c/17289 (originally started by Quanlong for Qifan)
These are now running. Thanks, - LaszloG On Wed, Apr 14, 2021 at 1:25 PM Riza Suminto <riza.sumi...@cloudera.com> wrote: > Thank you, Laszlo, for bringing jenkins.impala.io back up! > > On Wed, Apr 14, 2021, 1:36 AM Laszlo Gaal <laszlo.g...@cloudera.com> > wrote: > > > I have restarted the server, but the current version seems a bit > unstable. > > I will monitor the server more closely, but expect the server to fail, > and > > do not > > hesitate to point out to any issues you see. > > > > Failure analysis: > > The Jenkins server process was terminated by the OOM-killer at about > 01:15 > > UTC. > > The memory shortage was triggered by an anomaly: the OOM-killer's log > > suggests that > > a build job was started on the Jenkins master instance instead of a newly > > allocated worker VM. > > This quickly exhausted all the available memory on the master, which > caused > > the Jenkin server > > to be killed along with the errant build process. > > > > This is a failure that happened earlier; occurrences seem to be specific > to > > Jenkins server > > and plugin versions. If it happens again, the server will be upgraded > to a > > new version > > (if one with a fix is available), or rolled back to an earlier, known > good > > version. > > > > Thank you for your patience while this issue is being ironed out; > > please don't hesitate to call attention to any problems you experience > > with jenkins.impala.io > > > > - LaszloG > > > > On Wed, Apr 14, 2021 at 9:52 AM Laszlo Gaal <laszlo.g...@cloudera.com> > > wrote: > > > > > Thanks for raising the issue; I'm checking. > > > > > > - LaszloG > > > > > > On Wed, Apr 14, 2021 at 6:09 AM fucun chu <im.fu...@gmail.com> wrote: > > > > > >> [image: image.png]The Impala public Jenkins server is unavailable! > > >> > > >> Quanlong Huang <huangquanl...@gmail.com> 于2021年4月11日周日 上午8:51写道： > > >> > > >>> Thanks Laszlo for handling this! > > >>> > > >>> On Sun, Apr 11, 2021 at 2:14 AM Laszlo Gaal < > laszlo.g...@cloudera.com> > > >>> wrote: > > >>> > > >>> > The service is available again. > > >>> > > > >>> > Thank you for your patience, > > >>> > > > >>> > - LaszloG > > >>> > > > >>> > On Sat, Apr 10, 2021 at 7:15 PM Laszlo Gaal < > > laszlo.g...@cloudera.com> > > >>> > wrote: > > >>> > > > >>> > > Hi, > > >>> > > > > >>> > > The Impala public Jenkins server is undergoing upgrade and > > >>> maintenance, > > >>> > > during which it will be restarted multiple times. > > >>> > > Please do not submit jobs even if the server seems to be > available; > > >>> it > > >>> > may > > >>> > > still get restarted unexpectedly. > > >>> > > > > >>> > > I'll send out the "all clear" signal when the maintenance period > is > > >>> over. > > >>> > > > > >>> > > Thank you, > > >>> > > > > >>> > > - LaszloG > > >>> > > > > >>> > > > >>> > > >> > > >
