On Mon, Feb 12, 2018 at 4:58 PM, Thanh Ha <thanh...@linuxfoundation.org> wrote:
> On Mon, Feb 12, 2018 at 4:31 PM, Thanh Ha <thanh...@linuxfoundation.org> > wrote: > >> On Mon, Feb 12, 2018 at 11:57 AM, Thanh Ha <thanh...@linuxfoundation.org> >> wrote: >> >>> On Fri, Feb 9, 2018 at 2:42 PM, Thanh Ha <thanh...@linuxfoundation.org> >>> wrote: >>> >>>> Hi Everyone, >>>> >>>> *What:* The Linux Foundation will be performing maintenance on the >>>> Nexus server to perform 2 tasks: >>>> >>>> 1. Upgrade Nexus to the latest security patchset >>>> >>>> 2. Move file systems around to manage the inodes issue we ran into last >>>> week >>>> >>>> *When:* Monday, February 12 @ 9:00 - 11:00 PST (17:00 - 19:00 Dec 13 >>>> UTC) >>>> >>>> *Why:* Due to the issues we've had with running out of inodes on our >>>> system we've been using a temporary mitigation while that we could sync >>>> files to a new partition. If not done the server will run out of inodes >>>> again very soon. We will cut over to the new partition on Monday once the >>>> syncing is complete. >>>> >>>> We will be using this maintenance window to also apply security patches >>>> to the Nexus system as we've been informed on some CVEs that need to be >>>> addressed. >>>> >>>> *Impact:* Jenkins will be put into shutdown mode while we perform >>>> maintenance on the Nexus system. During this time Gerrit will be available >>>> however jobs will be queued but not run. >>>> >>>> Notices will be posted to the mailing lists and in #opendaylight on >>>> Freenode at the start and end of the maintenance. >>>> >>>> Regards, >>>> Thanh >>>> >>> >>> Maintenance has started. Jenkins is put in shutdown mode and in progress >>> jobs will fail to transfer files to Nexus until maintenance is complete. >>> For these jobs please requeue them and they will run post maintenance. >>> >> >> Hi Everyone, >> >> Sorry for the much longer than anticipated maintenance window. It is now >> complete and Jenkins & Nexus service have been restored. >> >> Logs have not yet completed syncing to the new file system and we expect >> it to take probably another week or so. In the meantime we've linked up old >> logs to this link temporary: >> >> https://logs.opendaylight.org/old-logs/ >> >> If you are looking for logs from before the maintenance window please use >> the above link, they will slowly start appearing in the new file system as >> they get copied over. Any logs from after the maintenance window will >> appear in the usual path. >> > > Looks like Nexus blew up as soon as we opened the floodgates. We're > investigating... > Nexus ran out of memory and was throwing a bunch of: Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded We bumped the memory on the system and it's back up now. Regards, Thanh
_______________________________________________ infrastructure mailing list infrastructure@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/infrastructure
