[
https://issues.apache.org/jira/browse/BEAM-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yifan zou updated BEAM-7109:
----------------------------
Description:
Beam Jenkins constantly break due to some weird errors such as "Unable to
create new native thread". The recent build worker failure happened on
[apache-beam-jenkins-8|[https://builds.apache.org/computer/apache-beam-jenkins-8/builds]].
Checking the thread number on that VM shows:
Thread limit: kernel.pid_max = 32768
Actual used: 32411
Dumping the thread usage (see attachment) exposed thread leaking on some Python
tests. And based on the execution history of the jenkins-8, the
[beam_PreCommit_Portable_Python_Commit]
(https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit) is
suspicious. We ran this test multiple times on a plain node and observed that
some thread started by +_apache_beam.runners.worker.sdk_worker_main_+ were not
tear down after tests complete. The stale threads finally accumulated and ate
the VM kernel thread quota. __
was:
Beam Jenkins constantly break due to some weird errors such as "Unable to
create new native thread". The recent build worker failure happened on
[apache-beam-jenkins-8|[https://builds.apache.org/computer/apache-beam-jenkins-8/builds]].
Checking the thread number on that VM shows:
Thread limit: kernel.pid_max = 32768
Actual used: 32411
Dumping the thread usage (see attachment) exposed thread leaking on some Python
tests. And based on the execution history of the jenkins-8, the
[beam_PreCommit_Portable_Python_Commit]([https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/])
is suspicious. We ran this test multiple times on a plain node and observed
that some thread started by +_apache_beam.runners.worker.sdk_worker_main_+ were
not tear down after tests complete. The stale threads finally accumulated and
ate the VM kernel thread quota. __
> Thread leaking in Portable Python Precommit
> --------------------------------------------
>
> Key: BEAM-7109
> URL: https://issues.apache.org/jira/browse/BEAM-7109
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: yifan zou
> Assignee: Ankur Goenka
> Priority: Critical
> Attachments: threadDump.txt
>
>
> Beam Jenkins constantly break due to some weird errors such as "Unable to
> create new native thread". The recent build worker failure happened on
> [apache-beam-jenkins-8|[https://builds.apache.org/computer/apache-beam-jenkins-8/builds]].
> Checking the thread number on that VM shows:
> Thread limit: kernel.pid_max = 32768
> Actual used: 32411
>
> Dumping the thread usage (see attachment) exposed thread leaking on some
> Python tests. And based on the execution history of the jenkins-8, the
> [beam_PreCommit_Portable_Python_Commit]
> (https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit) is
> suspicious. We ran this test multiple times on a plain node and observed that
> some thread started by +_apache_beam.runners.worker.sdk_worker_main_+ were
> not tear down after tests complete. The stale threads finally accumulated and
> ate the VM kernel thread quota. __
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
