[
https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=38469#comment-38469
]
Evgheni Dereveanchin commented on OVIRT-2586:
---------------------------------------------
I planned to reboot Jenkins tonight but it was busy running pipelines that
aren't easy to cancel. As a result, Jenkins completely locked up in the morning
with the UI being completely unreachable and backend threads timing out in the
background. Had to restart it and it's now coming back up.
The monitoring plugin was partially responsive still during the outage and
showed the following info:
|Java memory used: |15,590 Mb / 16,384 Mb *Usage is near the maximum,
you may need to optimize or to reconfigure (-Xmx)|
|Nb of http sessions: |8 |
|Nb of active threads
(current http requests): |33 |
|System load |2.78 |
|% System CPU |17.17|
Almost all memory got exhausted which is likely caused by a memory leak in the
SSE-gateway plugin coinciding with a large number of CI jobs appearing in the
queue. Adding memory to Java will likely just delay the symptoms as the memory
leak is still there (see JENKINS-51057)
!usedMemory_year.png|thumbnail!
>From the yearly memory graph the leak started around May-June this year and
>intensified in November.
To confirm the exact root cause we may need some lower-level troubleshooting of
the Java process yet I am not familiar with how that's done. [~mwperina] maybe
you can assist with the info that can be gathered to identify the root cause?
> Jenkins terribly slow and unresponsive
> --------------------------------------
>
> Key: OVIRT-2586
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586
> Project: oVirt - virtualization made easy
> Issue Type: Outage
> Reporter: sbonazzo
> Assignee: Evgheni Dereveanchin
> Priority: Highest
> Attachments: usedMemory_year.png
>
>
> Hi,
> jenkins is terribly slow and becoming worse every day.
> I tried to gain some speed by adding 4 cores to the VM through engine-phx.
> It's a bit better but the real issue doesn't seem related to CPU power.
> Can anybody investigate?
> --
> SANDRO BONAZZOLA
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
> Red Hat EMEA <https://www.redhat.com/>
> [email protected]
> <https://red.ht/sig>
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100095)
_______________________________________________
Infra mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/K5ELTOPQG27734FGDNP5U3ALNASMAATM/
