Re: jenkins.impala.io downtime
Hi, Jenkins is up and running again. As usual, please let me know if you experience any problems. Thanks, - LaszloG On Sat, Aug 5, 2023 at 6:50 PM Laszlo Gaal wrote: > Hi, > > The Impala public Jenkins service will be taken offline shortly for > maintenance and upgrades. > Expected length of the outage is ~3 hours; mail will be sent when it is > available again. > > Thank you for your patience, > > - LaszloG >
Re: jenkins.impala.io downtime
The service is available again. Thank you for your patience, - LaszloG On Sat, Mar 11, 2023 at 3:16 PM Laszlo Gaal wrote: > Hi, > > The public Impala Jenkins instance will be taken offline shortly for > upgrades and maintenance. The outage will take about 4 hours, i.e. it will > end at 18:00 UTC (19:00 BUD, 10:00 Pacific). > Mail will be sent when the service is available again. > > Thank you for your understanding, > > - LaszloG >
Re: jenkins.impala.io downtime
The public Jenkins service is available again. Thank you for your patience, - LaszloG On Wed, Feb 15, 2023 at 2:47 PM Laszlo Gaal wrote: > Hi, > > I am taking the public Jenkins server offline for a short upgrade; it > should be back in ~15 minutes. > I'll send mail when the service is available again. > > Thank you for your understanding, > > - LaszloG >
Re: jenkins.impala.io downtime
The server is available again. Thank you for your patience, - LaszloG On Sat, Sep 10, 2022 at 1:38 PM Laszlo Gaal wrote: > Hi, > > The Impala upstream Jenkins service will shortly be taken offline for > maintenance and upgrades. > Mail will be sent when the service is available again. > > Thank you for your understanding, > > - LaszloG >
Re: jenkins.impala.io downtime
No worries. Thanks for handling this, Laszlo! Quanlong On Tue, Aug 17, 2021 at 3:01 AM Laszlo Gaal wrote: > Hi, > > The maintenance is over, the service is up and running again. > > My apologies for the longer than usual outage, > > - LaszloG > > On Mon, Aug 16, 2021 at 3:05 PM Laszlo Gaal > wrote: > > > Hi, > > > > The Impala public Jenkins instance will be taken down for maintenance and > > upgrades. > > Planned duration is about 3 hours; mail will be sent when the service is > > available again. > > > > Thank you for your patience, > > > > - LaszloG > > >
Re: jenkins.impala.io downtime
Hi, The maintenance is over, the service is up and running again. My apologies for the longer than usual outage, - LaszloG On Mon, Aug 16, 2021 at 3:05 PM Laszlo Gaal wrote: > Hi, > > The Impala public Jenkins instance will be taken down for maintenance and > upgrades. > Planned duration is about 3 hours; mail will be sent when the service is > available again. > > Thank you for your patience, > > - LaszloG >
Re: jenkins.impala.io downtime
I forgot to add that I also restarted three GVDs that were interrupted by the outage: - https://gerrit.cloudera.org/#/c/17314 (originally started by Joe for John Sherman) - https://gerrit.cloudera.org/#/c/17252 (originally started by Joe for Qifan) - https://gerrit.cloudera.org/#/c/17289 (originally started by Quanlong for Qifan) These are now running. Thanks, - LaszloG On Wed, Apr 14, 2021 at 1:25 PM Riza Suminto wrote: > Thank you, Laszlo, for bringing jenkins.impala.io back up! > > On Wed, Apr 14, 2021, 1:36 AM Laszlo Gaal > wrote: > > > I have restarted the server, but the current version seems a bit > unstable. > > I will monitor the server more closely, but expect the server to fail, > and > > do not > > hesitate to point out to any issues you see. > > > > Failure analysis: > > The Jenkins server process was terminated by the OOM-killer at about > 01:15 > > UTC. > > The memory shortage was triggered by an anomaly: the OOM-killer's log > > suggests that > > a build job was started on the Jenkins master instance instead of a newly > > allocated worker VM. > > This quickly exhausted all the available memory on the master, which > caused > > the Jenkin server > > to be killed along with the errant build process. > > > > This is a failure that happened earlier; occurrences seem to be specific > to > > Jenkins server > > and plugin versions. If it happens again, the server will be upgraded > to a > > new version > > (if one with a fix is available), or rolled back to an earlier, known > good > > version. > > > > Thank you for your patience while this issue is being ironed out; > > please don't hesitate to call attention to any problems you experience > > with jenkins.impala.io > > > > - LaszloG > > > > On Wed, Apr 14, 2021 at 9:52 AM Laszlo Gaal > > wrote: > > > > > Thanks for raising the issue; I'm checking. > > > > > > - LaszloG > > > > > > On Wed, Apr 14, 2021 at 6:09 AM fucun chu wrote: > > > > > >> [image: image.png]The Impala public Jenkins server is unavailable! > > >> > > >> Quanlong Huang 于2021年4月11日周日 上午8:51写道： > > >> > > >>> Thanks Laszlo for handling this! > > >>> > > >>> On Sun, Apr 11, 2021 at 2:14 AM Laszlo Gaal < > [email protected]> > > >>> wrote: > > >>> > > >>> > The service is available again. > > >>> > > > >>> > Thank you for your patience, > > >>> > > > >>> > - LaszloG > > >>> > > > >>> > On Sat, Apr 10, 2021 at 7:15 PM Laszlo Gaal < > > [email protected]> > > >>> > wrote: > > >>> > > > >>> > > Hi, > > >>> > > > > >>> > > The Impala public Jenkins server is undergoing upgrade and > > >>> maintenance, > > >>> > > during which it will be restarted multiple times. > > >>> > > Please do not submit jobs even if the server seems to be > available; > > >>> it > > >>> > may > > >>> > > still get restarted unexpectedly. > > >>> > > > > >>> > > I'll send out the "all clear" signal when the maintenance period > is > > >>> over. > > >>> > > > > >>> > > Thank you, > > >>> > > > > >>> > > - LaszloG > > >>> > > > > >>> > > > >>> > > >> > > >
Re: jenkins.impala.io downtime
Thank you, Laszlo, for bringing jenkins.impala.io back up! On Wed, Apr 14, 2021, 1:36 AM Laszlo Gaal wrote: > I have restarted the server, but the current version seems a bit unstable. > I will monitor the server more closely, but expect the server to fail, and > do not > hesitate to point out to any issues you see. > > Failure analysis: > The Jenkins server process was terminated by the OOM-killer at about 01:15 > UTC. > The memory shortage was triggered by an anomaly: the OOM-killer's log > suggests that > a build job was started on the Jenkins master instance instead of a newly > allocated worker VM. > This quickly exhausted all the available memory on the master, which caused > the Jenkin server > to be killed along with the errant build process. > > This is a failure that happened earlier; occurrences seem to be specific to > Jenkins server > and plugin versions. If it happens again, the server will be upgraded to a > new version > (if one with a fix is available), or rolled back to an earlier, known good > version. > > Thank you for your patience while this issue is being ironed out; > please don't hesitate to call attention to any problems you experience > with jenkins.impala.io > > - LaszloG > > On Wed, Apr 14, 2021 at 9:52 AM Laszlo Gaal > wrote: > > > Thanks for raising the issue; I'm checking. > > > > - LaszloG > > > > On Wed, Apr 14, 2021 at 6:09 AM fucun chu wrote: > > > >> [image: image.png]The Impala public Jenkins server is unavailable! > >> > >> Quanlong Huang 于2021年4月11日周日 上午8:51写道： > >> > >>> Thanks Laszlo for handling this! > >>> > >>> On Sun, Apr 11, 2021 at 2:14 AM Laszlo Gaal > >>> wrote: > >>> > >>> > The service is available again. > >>> > > >>> > Thank you for your patience, > >>> > > >>> > - LaszloG > >>> > > >>> > On Sat, Apr 10, 2021 at 7:15 PM Laszlo Gaal < > [email protected]> > >>> > wrote: > >>> > > >>> > > Hi, > >>> > > > >>> > > The Impala public Jenkins server is undergoing upgrade and > >>> maintenance, > >>> > > during which it will be restarted multiple times. > >>> > > Please do not submit jobs even if the server seems to be available; > >>> it > >>> > may > >>> > > still get restarted unexpectedly. > >>> > > > >>> > > I'll send out the "all clear" signal when the maintenance period is > >>> over. > >>> > > > >>> > > Thank you, > >>> > > > >>> > > - LaszloG > >>> > > > >>> > > >>> > >> >
Re: jenkins.impala.io downtime
I have restarted the server, but the current version seems a bit unstable. I will monitor the server more closely, but expect the server to fail, and do not hesitate to point out to any issues you see. Failure analysis: The Jenkins server process was terminated by the OOM-killer at about 01:15 UTC. The memory shortage was triggered by an anomaly: the OOM-killer's log suggests that a build job was started on the Jenkins master instance instead of a newly allocated worker VM. This quickly exhausted all the available memory on the master, which caused the Jenkin server to be killed along with the errant build process. This is a failure that happened earlier; occurrences seem to be specific to Jenkins server and plugin versions. If it happens again, the server will be upgraded to a new version (if one with a fix is available), or rolled back to an earlier, known good version. Thank you for your patience while this issue is being ironed out; please don't hesitate to call attention to any problems you experience with jenkins.impala.io - LaszloG On Wed, Apr 14, 2021 at 9:52 AM Laszlo Gaal wrote: > Thanks for raising the issue; I'm checking. > > - LaszloG > > On Wed, Apr 14, 2021 at 6:09 AM fucun chu wrote: > >> [image: image.png]The Impala public Jenkins server is unavailable! >> >> Quanlong Huang 于2021年4月11日周日 上午8:51写道： >> >>> Thanks Laszlo for handling this! >>> >>> On Sun, Apr 11, 2021 at 2:14 AM Laszlo Gaal >>> wrote: >>> >>> > The service is available again. >>> > >>> > Thank you for your patience, >>> > >>> > - LaszloG >>> > >>> > On Sat, Apr 10, 2021 at 7:15 PM Laszlo Gaal >>> > wrote: >>> > >>> > > Hi, >>> > > >>> > > The Impala public Jenkins server is undergoing upgrade and >>> maintenance, >>> > > during which it will be restarted multiple times. >>> > > Please do not submit jobs even if the server seems to be available; >>> it >>> > may >>> > > still get restarted unexpectedly. >>> > > >>> > > I'll send out the "all clear" signal when the maintenance period is >>> over. >>> > > >>> > > Thank you, >>> > > >>> > > - LaszloG >>> > > >>> > >>> >>
Re: jenkins.impala.io downtime
Thanks for raising the issue; I'm checking. - LaszloG On Wed, Apr 14, 2021 at 6:09 AM fucun chu wrote: > [image: image.png]The Impala public Jenkins server is unavailable! > > Quanlong Huang 于2021年4月11日周日 上午8:51写道： > >> Thanks Laszlo for handling this! >> >> On Sun, Apr 11, 2021 at 2:14 AM Laszlo Gaal >> wrote: >> >> > The service is available again. >> > >> > Thank you for your patience, >> > >> > - LaszloG >> > >> > On Sat, Apr 10, 2021 at 7:15 PM Laszlo Gaal >> > wrote: >> > >> > > Hi, >> > > >> > > The Impala public Jenkins server is undergoing upgrade and >> maintenance, >> > > during which it will be restarted multiple times. >> > > Please do not submit jobs even if the server seems to be available; it >> > may >> > > still get restarted unexpectedly. >> > > >> > > I'll send out the "all clear" signal when the maintenance period is >> over. >> > > >> > > Thank you, >> > > >> > > - LaszloG >> > > >> > >> >
Re: jenkins.impala.io downtime
[image: image.png]The Impala public Jenkins server is unavailable! Quanlong Huang 于2021年4月11日周日 上午8:51写道： > Thanks Laszlo for handling this! > > On Sun, Apr 11, 2021 at 2:14 AM Laszlo Gaal > wrote: > > > The service is available again. > > > > Thank you for your patience, > > > > - LaszloG > > > > On Sat, Apr 10, 2021 at 7:15 PM Laszlo Gaal > > wrote: > > > > > Hi, > > > > > > The Impala public Jenkins server is undergoing upgrade and maintenance, > > > during which it will be restarted multiple times. > > > Please do not submit jobs even if the server seems to be available; it > > may > > > still get restarted unexpectedly. > > > > > > I'll send out the "all clear" signal when the maintenance period is > over. > > > > > > Thank you, > > > > > > - LaszloG > > > > > >
Re: jenkins.impala.io downtime
Thanks Laszlo for handling this! On Sun, Apr 11, 2021 at 2:14 AM Laszlo Gaal wrote: > The service is available again. > > Thank you for your patience, > > - LaszloG > > On Sat, Apr 10, 2021 at 7:15 PM Laszlo Gaal > wrote: > > > Hi, > > > > The Impala public Jenkins server is undergoing upgrade and maintenance, > > during which it will be restarted multiple times. > > Please do not submit jobs even if the server seems to be available; it > may > > still get restarted unexpectedly. > > > > I'll send out the "all clear" signal when the maintenance period is over. > > > > Thank you, > > > > - LaszloG > > >
Re: jenkins.impala.io downtime
The service is available again. Thank you for your patience, - LaszloG On Sat, Apr 10, 2021 at 7:15 PM Laszlo Gaal wrote: > Hi, > > The Impala public Jenkins server is undergoing upgrade and maintenance, > during which it will be restarted multiple times. > Please do not submit jobs even if the server seems to be available; it may > still get restarted unexpectedly. > > I'll send out the "all clear" signal when the maintenance period is over. > > Thank you, > > - LaszloG >
Re: jenkins.impala.io downtime
Thanks for taking care of this! On Sat, Aug 1, 2020 at 11:45 AM Laszlo Gaal wrote: > The service is back up and running, > > Thank you, > > - LaszloG > > On Sat, Aug 1, 2020 at 6:33 PM Laszlo Gaal > wrote: > > > The Impala public Jenkins instance will be taken offline for scheduled > > backup and maintenance. The service will return at about 11:30am Pacific > > time, which is 6:30pm UTC. > > > > Thank you for your patience, > > > > - LaszloG > > >
Re: jenkins.impala.io downtime
The service is back up and running, Thank you, - LaszloG On Sat, Aug 1, 2020 at 6:33 PM Laszlo Gaal wrote: > The Impala public Jenkins instance will be taken offline for scheduled > backup and maintenance. The service will return at about 11:30am Pacific > time, which is 6:30pm UTC. > > Thank you for your patience, > > - LaszloG >
Re: jenkins.impala.io downtime
Thank you for taking care of this, David! - LaszloG On Wed, Mar 11, 2020 at 5:09 AM David Rorke wrote: > The jenkins.impala.io upgrade is complete and the server is back online. > > On Tue, Mar 10, 2020 at 6:22 PM David Rorke wrote: > > > The public Impala Jenkins server will be going down for maintenance > > shortly. Another email will be sent when it becomes available again. > > >
Re: jenkins.impala.io downtime
The jenkins.impala.io upgrade is complete and the server is back online. On Tue, Mar 10, 2020 at 6:22 PM David Rorke wrote: > The public Impala Jenkins server will be going down for maintenance > shortly. Another email will be sent when it becomes available again. >
Re: jenkins.impala.io downtime
Thank Laszlo for handling this! Quanlong On Sun, Feb 16, 2020 at 1:24 AM Laszlo Gaal wrote: > The service is back online. > > Thank you for your patience, > >- LaszloG > > On Sat, Feb 15, 2020 at 4:25 PM Laszlo Gaal > wrote: > > > The public Impala Jenkins server will shortly go down for a short > > maintenance window. > > The server is currently quiet; mail will be sent when it becomes > available > > again. > > > > Thank you, > > > > - LaszloG > > >
Re: jenkins.impala.io downtime
The service is back online. Thank you for your patience, - LaszloG On Sat, Feb 15, 2020 at 4:25 PM Laszlo Gaal wrote: > The public Impala Jenkins server will shortly go down for a short > maintenance window. > The server is currently quiet; mail will be sent when it becomes available > again. > > Thank you, > > - LaszloG >
