On Mon, Feb 12, 2018 at 4:58 PM, Thanh Ha <thanh...@linuxfoundation.org>

> On Mon, Feb 12, 2018 at 4:31 PM, Thanh Ha <thanh...@linuxfoundation.org>
> wrote:
>> On Mon, Feb 12, 2018 at 11:57 AM, Thanh Ha <thanh...@linuxfoundation.org>
>> wrote:
>>> On Fri, Feb 9, 2018 at 2:42 PM, Thanh Ha <thanh...@linuxfoundation.org>
>>> wrote:
>>>> Hi Everyone,
>>>> *What:* The Linux Foundation will be performing maintenance on the
>>>> Nexus server to perform 2 tasks:
>>>> 1. Upgrade Nexus to the latest security patchset
>>>> 2. Move file systems around to manage the inodes issue we ran into last
>>>> week
>>>> *When:* Monday, February 12 @ 9:00 - 11:00 PST (17:00 - 19:00 Dec 13
>>>> UTC)
>>>> *Why:* Due to the issues we've had with running out of inodes on our
>>>> system we've been using a temporary mitigation while that we could sync
>>>> files to a new partition. If not done the server will run out of inodes
>>>> again very soon. We will cut over to the new partition on Monday once the
>>>> syncing is complete.
>>>> We will be using this maintenance window to also apply security patches
>>>> to the Nexus system as we've been informed on some CVEs that need to be
>>>> addressed.
>>>> *Impact:* Jenkins will be put into shutdown mode while we perform
>>>> maintenance on the Nexus system. During this time Gerrit will be available
>>>> however jobs will be queued but not run.
>>>> Notices will be posted to the mailing lists and in #opendaylight on
>>>> Freenode at the start and end of the maintenance.
>>>> Regards,
>>>> Thanh
>>> Maintenance has started. Jenkins is put in shutdown mode and in progress
>>> jobs will fail to transfer files to Nexus until maintenance is complete.
>>> For these jobs please requeue them and they will run post maintenance.
>> Hi Everyone,
>> Sorry for the much longer than anticipated maintenance window. It is now
>> complete and Jenkins & Nexus service have been restored.
>> Logs have not yet completed syncing to the new file system and we expect
>> it to take probably another week or so. In the meantime we've linked up old
>> logs to this link temporary:
>> https://logs.opendaylight.org/old-logs/
>> If you are looking for logs from before the maintenance window please use
>> the above link, they will slowly start appearing in the new file system as
>> they get copied over. Any logs from after the maintenance window will
>> appear in the usual path.
> Looks like Nexus blew up as soon as we opened the floodgates. We're
> investigating...

Nexus ran out of memory and was throwing a bunch of: Caused by:
java.lang.OutOfMemoryError: GC overhead limit exceeded

We bumped the memory on the system and it's back up now.

infrastructure mailing list

Reply via email to