Re: [OpenDaylight Infrastructure] [integration-dev] ODL Nexus Maintenance 2018-02-12 @ 9:00 - 11:00 PST (17:00 - 19:00 Feb 12 UTC)

2018-02-12 Thread Thanh Ha 
Look in the old-logs directory for logs from before the switch. It is still
being synced over so won't appear in the usual path until the sync is
complete (we expect this to take probably another week). But the symlink
"old-logs" points to the old partition and the full data is still there so
if you take the log path for an old log and insert "old-logs" to the path
just before "releng" you should be able to see the logs.

This is just temporary until rsync completes syncing all the files over.
Since logs is a bunch of very small files this will take a very long time
to copy.

Hope this helps,
Thanh


On Mon, Feb 12, 2018 at 6:14 PM, Luis Gomez  wrote:

> Hi Thanh, I see nexus is back but not the files stored in
> logs.opendaylight.org for all our builds. Have we lost that?
>
> On Feb 12, 2018, at 2:16 PM, Thanh Ha 
> wrote:
>
> On Mon, Feb 12, 2018 at 4:58 PM, Thanh Ha 
> wrote:
>
>> On Mon, Feb 12, 2018 at 4:31 PM, Thanh Ha 
>> wrote:
>>
>>> On Mon, Feb 12, 2018 at 11:57 AM, Thanh Ha >> > wrote:
>>>
 On Fri, Feb 9, 2018 at 2:42 PM, Thanh Ha 
 wrote:

> Hi Everyone,
>
> *What:* The Linux Foundation will be performing maintenance on the
> Nexus server to perform 2 tasks:
>
> 1. Upgrade Nexus to the latest security patchset
>
> 2. Move file systems around to manage the inodes issue we ran into
> last week
>
> *When:* Monday, February 12 @ 9:00 - 11:00 PST (17:00 - 19:00 Dec 13
> UTC)
>
> *Why:* Due to the issues we've had with running out of inodes on our
> system we've been using a temporary mitigation while that we could sync
> files to a new partition. If not done the server will run out of inodes
> again very soon. We will cut over to the new partition on Monday once the
> syncing is complete.
>
> We will be using this maintenance window to also apply security
> patches to the Nexus system as we've been informed on some CVEs that need
> to be addressed.
>
> *Impact:* Jenkins will be put into shutdown mode while we perform
> maintenance on the Nexus system. During this time Gerrit will be available
> however jobs will be queued but not run.
>
> Notices will be posted to the mailing lists and in #opendaylight on
> Freenode at the start and end of the maintenance.
>
> Regards,
> Thanh
>

 Maintenance has started. Jenkins is put in shutdown mode and in
 progress jobs will fail to transfer files to Nexus until maintenance is
 complete. For these jobs please requeue them and they will run post
 maintenance.

>>>
>>> Hi Everyone,
>>>
>>> Sorry for the much longer than anticipated maintenance window. It is now
>>> complete and Jenkins & Nexus service have been restored.
>>>
>>> Logs have not yet completed syncing to the new file system and we expect
>>> it to take probably another week or so. In the meantime we've linked up old
>>> logs to this link temporary:
>>>
>>> https://logs.opendaylight.org/old-logs/
>>>
>>> If you are looking for logs from before the maintenance window please
>>> use the above link, they will slowly start appearing in the new file system
>>> as they get copied over. Any logs from after the maintenance window will
>>> appear in the usual path.
>>>
>>
>> Looks like Nexus blew up as soon as we opened the floodgates. We're
>> investigating...
>>
>
> Nexus ran out of memory and was throwing a bunch of: Caused by:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> We bumped the memory on the system and it's back up now.
>
> Regards,
> Thanh
>
> ___
> integration-dev mailing list
> integration-...@lists.opendaylight.org
> https://lists.opendaylight.org/mailman/listinfo/integration-dev
>
>
>
___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure

Re: [OpenDaylight Infrastructure] [integration-dev] ODL Nexus Maintenance 2018-02-12 @ 9:00 - 11:00 PST (17:00 - 19:00 Feb 12 UTC)

2018-02-12 Thread Luis Gomez 
Hi Thanh, I see nexus is back but not the files stored in logs.opendaylight.org 
for all our builds. Have we lost that?

> On Feb 12, 2018, at 2:16 PM, Thanh Ha  wrote:
> 
> On Mon, Feb 12, 2018 at 4:58 PM, Thanh Ha  > wrote:
> On Mon, Feb 12, 2018 at 4:31 PM, Thanh Ha  > wrote:
> On Mon, Feb 12, 2018 at 11:57 AM, Thanh Ha  > wrote:
> On Fri, Feb 9, 2018 at 2:42 PM, Thanh Ha  > wrote:
> Hi Everyone,
> 
> What: The Linux Foundation will be performing maintenance on the Nexus server 
> to perform 2 tasks:
> 
> 1. Upgrade Nexus to the latest security patchset
> 
> 2. Move file systems around to manage the inodes issue we ran into last week
> 
> When: Monday, February 12 @ 9:00 - 11:00 PST (17:00 - 19:00 Dec 13 UTC)
> 
> Why: Due to the issues we've had with running out of inodes on our system 
> we've been using a temporary mitigation while that we could sync files to a 
> new partition. If not done the server will run out of inodes again very soon. 
> We will cut over to the new partition on Monday once the syncing is complete. 
> 
> We will be using this maintenance window to also apply security patches to 
> the Nexus system as we've been informed on some CVEs that need to be 
> addressed.
> 
> Impact: Jenkins will be put into shutdown mode while we perform maintenance 
> on the Nexus system. During this time Gerrit will be available however jobs 
> will be queued but not run.
> 
> Notices will be posted to the mailing lists and in #opendaylight on
> Freenode at the start and end of the maintenance.
> 
> Regards,
> Thanh
> 
> Maintenance has started. Jenkins is put in shutdown mode and in progress jobs 
> will fail to transfer files to Nexus until maintenance is complete. For these 
> jobs please requeue them and they will run post maintenance.
> 
> Hi Everyone,
> 
> Sorry for the much longer than anticipated maintenance window. It is now 
> complete and Jenkins & Nexus service have been restored.
> 
> Logs have not yet completed syncing to the new file system and we expect it 
> to take probably another week or so. In the meantime we've linked up old logs 
> to this link temporary:
> 
> https://logs.opendaylight.org/old-logs/ 
> 
> 
> If you are looking for logs from before the maintenance window please use the 
> above link, they will slowly start appearing in the new file system as they 
> get copied over. Any logs from after the maintenance window will appear in 
> the usual path.
> 
> Looks like Nexus blew up as soon as we opened the floodgates. We're 
> investigating...
> 
> Nexus ran out of memory and was throwing a bunch of: Caused by: 
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 
> We bumped the memory on the system and it's back up now.
> 
> Regards,
> Thanh
> 
> ___
> integration-dev mailing list
> integration-...@lists.opendaylight.org
> https://lists.opendaylight.org/mailman/listinfo/integration-dev

___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure